How to POST a fine-tuning job to OpenAI API using the correct file format? – Rest

The Problem:

An individual is facing difficulties in using the OpenAI API for Fine-tuning. They have a file in prompt-completion format and wish to utilize it for fine-tuning a model, but the API requires the data to be in chat-completion format. Despite attempts to convert the data and upload it, they consistently receive an error message about invalid file format. The task is to provide a fully working example using curl commands to successfully upload a file in the correct format and initiate a Fine-tuning job.

The Solutions:

Solution 1: Using Chat Completion Format for Fine-tuning

1. Prepare the Training Data:

• Ensure your training data is in the chat completion format, which resembles a conversation between a user and an assistant.
• Each entry should include messages from both the "user" and "assistant" roles.
• Make sure the file is in JSONL format, with each entry on a separate line.

2. Upload the Training File:

• Use curl to upload your training file to OpenAI.
• Set the purpose parameter to "fine-tune".

curl -X POST https://api.openai.com/v1/files \
-H 'authorization: Bearer <token>' \
-F file=@chat_completion_data.jsonl \
-F purpose=fine-tune

3. Create a Fine-tuning Job:

• Once the file is processed, use curl again to create a fine-tuning job.
• Specify the training file ID and the model you want to fine-tune.

curl -X POST https://api.openai.com/v1/fine_tuning/jobs \
-H 'authorization: Bearer <token>' \
-H 'Content-Type: application/json' \
-d '{
  "training_file": "<training_file_id>",
  "model": "<model_id>"
}'

Replace <training_file_id> with the ID of the uploaded training file and <model_id> with the ID of the model you want to fine-tune, such as "gpt-3.5-turbo".

4. Monitor the Fine-tuning Job:

• Use the OpenAI API to check the status of your fine-tuning job.
• Once the job is complete, you can download the fine-tuned model.

Solution 2: Using the correct JSONL file format for OpenAI’s gpt-3.5-turbo-0613 model

To successfully POST a fine-tuning job to the OpenAI API using the correct file format, follow these steps:

Step 1: Ensure the Correct File Format

Ensure your JSONL file is in the correct format required by the gpt-3.5-turbo-0613 model. Here’s an example:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a knowledgeable assistant."
    },
    {
      "role": "user",
      "content": "What is Laravel?"
    },
    {
      "role": "assistant",
      "content": "Laravel is a free, open-source PHP web framework."
    }
  ]
}

Each JSON object in the file represents a single prompt and answer identification. It includes three objects:

• system role and its content (system’s message)
• user role and their content (user’s question or prompt)
• assistant role and its content (answer or completion to the user’s question)

Step 2: Create a Fine-Tuning Job

Create a new fine-tuning job by sending a POST request to the OpenAI API with the following information:

POST https://api.openai.com/v1/fine_tuning/jobs

{
  "training_file": "file-2FLvF7VJSBsEOFSwJAf1XXXX",
  "model": "gpt-3.5-turbo-0613"
}

Make sure to replace "file-2FLvF7VJSBsEOFSwJAf1XXXX" with the ID of your file containing the data in the correct format.

Step 3: Monitor the Job’s Progress

Once the job is created, you can monitor its progress by sending a GET request to the API with the job’s ID:

GET https://api.openai.com/v1/fine_tuning/jobs/{job_id}

Replace "" with the actual ID of your fine-tuning job.

Step 4: Use the Fine-Tuned Model

After the fine-tuning job is complete, you can use the fine-tuned model to generate text. Send a POST request to the API with the following information:

POST https://api.openai.com/v1/models/{model_id}/generate

{
  "prompt": {
    "text": "Generate a story about a brave knight."
  }
}

Replace "" with the ID of your fine-tuned model.

Following these steps with the correct JSONL file format should allow you to successfully POST a fine-tuning job to OpenAI’s API and use the fine-tuned model for text generation.

Video Explanation: