Skip to content

Add editing functionality BEN-1079 #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 13, 2025
Merged

Add editing functionality BEN-1079 #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 57 additions & 13 deletions app/api/generate/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
// app/api/generate/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { generateApp } from '@/lib/openai';
import { generateApp, editApp } from '@/lib/openai';
import { createSandbox } from '@/lib/e2b';
import { componentSchema } from '@/lib/schemas';
import { Benchify } from 'benchify';
import { applyPatch } from 'diff';
import { z } from 'zod';
import { benchifyFileSchema } from '@/lib/schemas';

const benchify = new Benchify({
apiKey: process.env.BENCHIFY_API_KEY,
Expand Down Expand Up @@ -32,12 +34,30 @@ export default App;`
}
];

// Extended schema to support editing
const extendedComponentSchema = componentSchema.extend({
existingFiles: benchifyFileSchema.optional(),
editInstruction: z.string().optional(),
});

// Helper function to merge updated files with existing files
function mergeFiles(existingFiles: z.infer<typeof benchifyFileSchema>, updatedFiles: z.infer<typeof benchifyFileSchema>): z.infer<typeof benchifyFileSchema> {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Ensure Uniqueness of File Paths in Result

The returned array should ensure uniqueness of files by their path, preventing any duplicate paths.

Outcome Example Input # Inputs % of Total
✅ superjson.parse('{"json":[[[{"path":"aaaaXaaamo... view full input 200 100.0%

view all inputs
Your property-based test has passed! This test ensures that the mergeFiles function returns an array of files with unique paths. The test generated an example with two arrays of files, existingFiles and updatedFiles, and successfully verified that the merged result contains no duplicate paths. The test's success indicates that the mergeFiles function is correctly handling file merging and maintaining path uniqueness.

Unit Tests 
// Unit Test for "Ensure Uniqueness of File Paths in Result": The returned array should ensure uniqueness of files by their path, preventing any duplicate paths.
function benchify_s(s) {
    return s.replace(/[^a-zA-Z0-9]/g, 'a');
}

it('benchify_s_exec_test_passing_0', () => {
  const args = superjson.parse(
    '{"json":[[[{"path":"aaaaXaaamoa","content":"caller"}],[{"path":"MaFaaaYaXT","content":"sFaM"}]]]}',
  );

  benchify_s(...args);
});

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Implement Correct Override Logic for Merged Files

When merging files, updatedFiles should overwrite existingFiles based on matching path elements.

Outcome Example Input # Inputs % of Total
✅ superjson.parse('{"json":[[[{"path":"7e22ca07-0... view full input 200 100.0%

view all inputs
The property-based test has passed, indicating that the mergeFiles function correctly overwrites existing files with updated files based on matching path elements. The test successfully merged the provided existingFiles and updatedFiles arrays, and the resulting merged files contained the expected content from the updatedFiles array. This suggests that the overriding logic in the mergeFiles function is working as intended.

Unit Tests 
// Unit Test for "Implement Correct Override Logic for Merged Files": When merging files, updatedFiles should overwrite existingFiles based on matching path elements.
function benchify_s(s) {
    return s.replace(/[^a-zA-Z0-9]/g, 'a');
}

it('benchify_s_exec_test_passing_0', () => {
  const args = superjson.parse(
    '{"json":[[[{"path":"7e22ca07-0465-384d-88f4-c6e4a53ef4b6","content":"aaaFaa"}],[{"path":"d7cc1abd-8665-255a-a617-417238bcfa92","content":"Wa"},{"path":"9f250b89-56ff-18b1-9a8b-5734c0f8370a","content":"aIaag"},{"path":"113cc398-cd5b-50ed-8539-a5c522a97cfd","content":"xegaadaaCz1"}]]]}',
  );

  benchify_s(...args);
});

const existingMap = new Map(existingFiles.map(file => [file.path, file]));

// Apply updates
updatedFiles.forEach(updatedFile => {
existingMap.set(updatedFile.path, updatedFile);
});

return Array.from(existingMap.values());
}

export async function POST(request: NextRequest) {
try {
const body = await request.json();

// Validate the request using Zod schema
const validationResult = componentSchema.safeParse(body);
// Validate the request using extended schema
const validationResult = extendedComponentSchema.safeParse(body);

if (!validationResult.success) {
return NextResponse.json(
Expand All @@ -46,30 +66,53 @@ export async function POST(request: NextRequest) {
);
}

const { description } = validationResult.data;
const { description, existingFiles, editInstruction } = validationResult.data;

console.log('API Request:', {
isEdit: !!(existingFiles && editInstruction),
filesCount: existingFiles?.length || 0,
editInstruction: editInstruction || 'none',
description: description || 'none'
});

let filesToSandbox;

// Determine if this is an edit request or new generation
if (existingFiles && editInstruction) {
// Edit existing code
console.log('Processing edit request...');
console.log('Existing files:', existingFiles.map(f => ({ path: f.path, contentLength: f.content.length })));

const updatedFiles = await editApp(existingFiles, editInstruction);
console.log('Updated files from AI:', updatedFiles.map(f => ({ path: f.path, contentLength: f.content.length })));

// Generate the Vue app using OpenAI
let generatedFiles;
if (debug) {
generatedFiles = buggyCode;
// Merge the updated files with the existing files
filesToSandbox = mergeFiles(existingFiles, updatedFiles);
console.log('Final merged files:', filesToSandbox.map(f => ({ path: f.path, contentLength: f.content.length })));
} else {
generatedFiles = await generateApp(description);
// Generate new app
console.log('Processing new generation request...');
if (debug) {
filesToSandbox = buggyCode;
} else {
filesToSandbox = await generateApp(description);
}
}

// Repair the generated code using Benchify's API
// const { data } = await benchify.fixer.run({
// files: generatedFiles.map(file => ({
// files: filesToSandbox.map(file => ({
// path: file.path,
// contents: file.content
// }))
// });

let repairedFiles = generatedFiles;
let repairedFiles = filesToSandbox;
// if (data) {
// const { success, diff } = data;

// if (success && diff) {
// repairedFiles = generatedFiles.map(file => {
// repairedFiles = filesToSandbox.map(file => {
// const patchResult = applyPatch(file.content, diff);
// return {
// ...file,
Expand All @@ -83,12 +126,13 @@ export async function POST(request: NextRequest) {

// Return the results to the client
return NextResponse.json({
originalFiles: generatedFiles,
originalFiles: filesToSandbox,
repairedFiles: sandboxResult.allFiles, // Use the allFiles from the sandbox
buildOutput: `Sandbox created with template: ${sandboxResult.template}, ID: ${sandboxResult.sbxId}`,
previewUrl: sandboxResult.url,
buildErrors: sandboxResult.buildErrors,
hasErrors: sandboxResult.hasErrors,
...(editInstruction && { editInstruction }),
});
} catch (error) {
console.error('Error generating app:', error);
Expand Down
7 changes: 6 additions & 1 deletion app/chat/page.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,12 @@ export default function ChatPage() {
<div className="w-1/4 min-w-80 border-r border-border bg-card flex-shrink-0">
<ChatInterface
initialPrompt={initialPrompt}
onUpdateResult={setResult}
currentFiles={result?.repairedFiles || result?.originalFiles}
onUpdateResult={(updatedResult) => {
setResult(updatedResult);
// Save updated result to sessionStorage
sessionStorage.setItem('builderResult', JSON.stringify(updatedResult));
}}
/>
</div>

Expand Down
68 changes: 55 additions & 13 deletions components/ui-builder/chat-interface.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,24 @@ interface Message {

interface ChatInterfaceProps {
initialPrompt: string;
currentFiles?: z.infer<typeof benchifyFileSchema>;
onUpdateResult: (result: {
repairedFiles?: z.infer<typeof benchifyFileSchema>;
originalFiles?: z.infer<typeof benchifyFileSchema>;
buildOutput: string;
previewUrl: string;
buildErrors?: Array<{
type: 'typescript' | 'build' | 'runtime';
message: string;
file?: string;
line?: number;
column?: number;
}>;
hasErrors?: boolean;
}) => void;
}

export function ChatInterface({ initialPrompt, onUpdateResult }: ChatInterfaceProps) {
export function ChatInterface({ initialPrompt, currentFiles, onUpdateResult }: ChatInterfaceProps) {
const [messages, setMessages] = useState<Message[]>([
{
id: '1',
Expand Down Expand Up @@ -64,30 +73,63 @@ export function ChatInterface({ initialPrompt, onUpdateResult }: ChatInterfacePr
};

setMessages(prev => [...prev, userMessage]);
const editInstruction = newMessage;
setNewMessage('');
setIsLoading(true);

try {
// Here you would call your API to process the new request
// For now, we'll add a placeholder response
const assistantMessage: Message = {
// Add thinking message
const thinkingMessage: Message = {
id: (Date.now() + 1).toString(),
type: 'assistant',
content: "I understand your request. Let me update the component for you...",
timestamp: new Date(),
};

setMessages(prev => [...prev, assistantMessage]);

// TODO: Implement actual regeneration with the new prompt
// This would call your generate API with the conversation context
setMessages(prev => [...prev, thinkingMessage]);

// Call the edit API
const response = await fetch('/api/generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
type: 'component',
description: '', // Not used for edits
existingFiles: currentFiles,
editInstruction: editInstruction,
}),
});

console.log('Edit request:', {
existingFiles: currentFiles,
editInstruction: editInstruction,
filesCount: currentFiles?.length || 0
});

if (!response.ok) {
throw new Error('Failed to edit component');
}

const editResult = await response.json();
console.log('Edit response:', editResult);

// Update the result in the parent component
onUpdateResult(editResult);

// Update the thinking message to success
setMessages(prev => prev.map(msg =>
msg.id === thinkingMessage.id
? { ...msg, content: `Great! I've updated the component according to your request: "${editInstruction}"` }
: msg
));

} catch (error) {
console.error('Error processing message:', error);
console.error('Error processing edit:', error);
const errorMessage: Message = {
id: (Date.now() + 1).toString(),
type: 'assistant',
content: "I'm sorry, there was an error processing your request. Please try again.",
content: "I'm sorry, there was an error processing your edit request. Please try again.",
timestamp: new Date(),
};
setMessages(prev => [...prev, errorMessage]);
Expand Down Expand Up @@ -137,8 +179,8 @@ export function ChatInterface({ initialPrompt, onUpdateResult }: ChatInterfacePr
>
<div
className={`max-w-[80%] p-3 rounded-lg ${message.type === 'user'
? 'bg-primary text-primary-foreground'
: 'bg-muted text-muted-foreground'
? 'bg-primary text-primary-foreground'
: 'bg-muted text-muted-foreground'
}`}
>
<p className="text-sm whitespace-pre-wrap">{message.content}</p>
Expand Down
44 changes: 42 additions & 2 deletions lib/openai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
import { streamObject } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';
import { REACT_APP_SYSTEM_PROMPT, REACT_APP_USER_PROMPT, TEMPERATURE, MODEL } from './prompts';
import { REACT_APP_SYSTEM_PROMPT, REACT_APP_USER_PROMPT, TEMPERATURE, MODEL, EDIT_SYSTEM_PROMPT, createEditUserPrompt } from './prompts';
import { benchifyFileSchema } from './schemas';

const OPENAI_API_KEY = process.env.OPENAI_API_KEY;

Expand All @@ -16,7 +17,9 @@ const fileSchema = z.object({
content: z.string()
});

// Generate a Vue application using AI SDK


// Generate a new application using AI SDK
export async function generateApp(
description: string,
): Promise<Array<{ path: string; content: string }>> {
Expand Down Expand Up @@ -50,4 +53,41 @@ export async function generateApp(
console.error('Error generating app:', error);
throw error;
}
}

// Edit existing application using AI SDK
export async function editApp(
existingFiles: z.infer<typeof benchifyFileSchema>,
editInstruction: string,
): Promise<Array<{ path: string; content: string }>> {
console.log("Editing app with instruction: ", editInstruction);

try {
const { elementStream } = streamObject({
model: openai('gpt-4o-mini'),
output: 'array',
schema: fileSchema,
temperature: 0.3, // Lower temperature for more consistent edits
messages: [
{ role: 'system', content: EDIT_SYSTEM_PROMPT },
{ role: 'user', content: createEditUserPrompt(existingFiles, editInstruction) }
]
});

const updatedFiles = [];
for await (const file of elementStream) {
updatedFiles.push(file);
}

if (!updatedFiles.length) {
throw new Error("Failed to generate updated files - received empty response");
}

console.log("Generated updated files: ", updatedFiles);

return updatedFiles;
} catch (error) {
console.error('Error editing app:', error);
throw error;
}
}
36 changes: 36 additions & 0 deletions lib/prompts.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
// lib/prompts.ts
import { z } from 'zod';
import { benchifyFileSchema } from "./schemas";

export const REACT_APP_SYSTEM_PROMPT = `You are an expert React, TypeScript, and Tailwind CSS developer.
You will be generating React application code based on the provided description.
Expand Down Expand Up @@ -47,5 +49,39 @@ export const REACT_APP_USER_PROMPT = (description: string) => `
Create a React application with the following requirements:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Output Format Consistency

The function should return a string that starts with 'Create a React application with the following requirements:' followed by the input 'description'.

Outcome Example Input # Inputs % of Total
✅ superjson.parse('{"json":[["naasa"]]}')... view full input 200 100.0%

view all inputs
The test has passed successfully! The input argument ["{\"json\":[[\"naasa\"]]}"] was processed correctly by the REACT_APP_USER_PROMPT function, which returned a string starting with "Create a React application with the following requirements:" followed by the input description, as expected.

Unit Tests 
// Unit Test for "Output Format Consistency": The function should return a string that starts with 'Create a React application with the following requirements:' followed by the input 'description'.
function benchify_s(s) {
    return s.replace(/[^a-zA-Z0-9]/g, 'a');
}

it('benchify_s_exec_test_passing_0', () => {
  const args = superjson.parse('{"json":[["naasa"]]}');

  benchify_s(...args);
});

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Description Must Be a String

The function should accept only string inputs for the 'description' parameter.

Outcome Example Input # Inputs % of Total
✅ superjson.parse('{"json":[["\n-2~eX9(l"]]}')...` view full input 200 100.0%

view all inputs
The property-based test has passed successfully. This means that the REACT_APP_USER_PROMPT function correctly handles string inputs for the 'description' parameter, as expected. In this specific test, the function was given the input "{\"json\":[[\"n-2~eX9(l"]]}"` and returned a defined result.

Unit Tests 
// Unit Test for "Description Must Be a String": The function should accept only string inputs for the 'description' parameter.
function benchify_description(description) {
    const result = REACT_APP_USER_PROMPT(description);
    expect(result).toBeDefined();
}

it('benchify_description_exec_test_passing_0', () => {
  const args = superjson.parse('{"json":[["`n-2~eX9(l"]]}');

  benchify_description(...args);
});

${description}`;

export const EDIT_SYSTEM_PROMPT = `You are an expert React/TypeScript developer. You will be given existing code files and an edit instruction. Your job is to modify the existing code according to the instruction while maintaining:

1. Code quality and best practices
2. Existing functionality that shouldn't be changed
3. Proper TypeScript types
4. Modern React patterns
5. Tailwind CSS for styling
6. shadcn/ui components where appropriate

Return ONLY the files that need to be changed. Do not return unchanged files.

Rules:
- Only return files that have actual changes
- Make minimal changes necessary to fulfill the instruction
- Keep all imports and dependencies that are still needed
- Add new dependencies only if absolutely necessary
- Use Tailwind classes for styling changes
- Follow React best practices
- Ensure all returned files are complete and valid`;

export function createEditUserPrompt(files: z.infer<typeof benchifyFileSchema>, editInstruction: string): string {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Schema Adherence for Files

Ensure that each object in the 'files' array adheres to the 'benchifyFileSchema', containing at least a 'path' and 'content' as non-empty strings.

Outcome Example Input # Inputs % of Total
✅ superjson.parse('{"json":[[[{"path":"jlgN","con... view full input 200 100.0%

view all inputs
The test has passed successfully! The property test verified that the 'files' array conforming to the 'benchifyFileSchema' indeed contains objects with non-empty strings for 'path' and 'content'. The test example provided an array of 10 files with various paths and contents, and the test correctly validated each object in the array, ensuring they adhere to the expected schema.

Unit Tests 
// Unit Test for "Schema Adherence for Files": Ensure that each object in the 'files' array adheres to the 'benchifyFileSchema', containing at least a 'path' and 'content' as non-empty strings.
function benchify_files(files) {
    expect(files).toBeTruthy();
    files.forEach(file => {
        expect(file.path).toBeTruthy();
        expect(file.content).toBeTruthy();
    });
}

it('benchify_files_exec_test_passing_0', () => {
  const args = superjson.parse(
    '{"json":[[[{"path":"jlgN","content":"HQA"},{"path":"#U","content":"uF"},{"path":"d\\"@52SBD","content":"^[O3Up{5Q"},{"path":"K -o\'Z@?a","content":"2:-~b&)e9l"},{"path":"5|","content":" pf!.oGd&6Y"},{"path":"(ygy%;","content":"b_qe"},{"path":"5j:q+kXm","content":"<7"},{"path":"0+DxMwyYq8(\\"","content":"Db JT"},{"path":"-F4zGt?W<","content":"B@{"},{"path":"j$**Q<;4E","content":"0S;%NEHzw"},{"path":"a!2<>-=_us3","content":"iI3&obq"}]]]}',
  );

  benchify_files(...args);
});

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Consistent Output Formatting

Ensure the output consistently starts with a specific introductory statement and concludes with a formatted edit instruction section.

Outcome Example Input # Inputs % of Total
❌ superjson.parse('{"json":[[[{"path":"goWAus)<4q... view full input 400 100.0%

view all inputs
The test is failing because the output of the createEditUserPrompt function does not consistently start with the introductory statement "Here are the current files:\n\n" for the given input. The input consists of an array of objects with "path" and "content" properties, and an edit instruction string. The error is an AssertionError, indicating that the expectation of the output starting with the introductory statement was not met.

Stack Trace 
Error: expect(received).toBe(expected)

Expected: true
Received: false

    at toBe (unknown)
    at <anonymous> (/app/repo/lib/pver_93591327-e049-4547-b394-b4ed47913f95.test.ts:63:73)
    at <anonymous> (/app/configuration/fc.setup.ts:183:11)
    at run (/app/node_modules/fast-check/lib/esm/check/property/Property.generic.js:46:33)
    at runIt (/app/node_modules/fast-check/lib/esm/check/runner/Runner.js:18:30)
    at check (/app/node_modules/fast-check/lib/esm/check/runner/Runner.js:62:11)
    at <anonymous> (/app/configuration/fc.setup.ts:197:14)
    at assertWithLogging (/app/configuration/fc.setup.ts:125:3)
    at <anonymous> (/app/repo/lib/pver_93591327-e049-4547-b394-b4ed47913f95.test.ts:39:6)
Unit Tests 
// Unit Test for "Consistent Output Formatting": Ensure the output consistently starts with a specific introductory statement and concludes with a formatted edit instruction section.
function benchify_files(files, editInstruction) {
    const output = createEditUserPrompt(files, editInstruction);
    expect(output.startsWith("Here are the current files:\n\n")).toBe(true);
    for (const file of files) {
        expect(output.includes(`### ${file.path}\n\`\`\`\n${file.content}\n\`\`\``)).toBe(true);
    }
    expect(output.endsWith(`Edit instruction: ${editInstruction}`)).toBe(true);
}

it('benchify_files_exec_test_failing_0', () => {
  const args = superjson.parse(
    '{"json":[[[{"path":"goWAus)<4qvV","content":"elb"},{"path":"fe%PMi76L>%;","content":"J,Ps&WKS"},{"path":"nY6c^o G-^","content":"l Y`5"},{"path":"]2X","content":"A64qi+z~"}],"+i+_et=R-|Z"]]}',
  );

  benchify_files(...args);
});

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Error Handling for Missing or Invalid Input

Handle cases where the 'files' array is empty or 'editInstruction' does not provide a valid string, to avoid malformed output.

Outcome Example Input # Inputs % of Total
❌ superjson.parse('{"json":[[[{"path":"(eN^%","co... view full input 400 100.0%

view all inputs
Here's a summary of the test results:

The test failed with a TypeError due to an unexpected usage of an instance of FrequencyArbitrary as a function. This issue occurred when processing an array of files with specific paths and contents, where the function createEditUserPrompt was expected to handle cases with an empty 'files' array or invalid 'editInstruction' strings. To resolve this, review the implementation of the createEditUserPrompt function and ensure it correctly handles these edge cases.

Stack Trace 
TypeError: p is not a function. (In 'p(...t)', 'p' is an instance of FrequencyArbitrary)
    at <anonymous> (/app/node_modules/fast-check/lib/esm/check/property/Property.js:14:54)
    at <anonymous> (/app/configuration/fc.setup.ts:183:11)
    at run (/app/node_modules/fast-check/lib/esm/check/property/Property.generic.js:46:33)
    at runIt (/app/node_modules/fast-check/lib/esm/check/runner/Runner.js:18:30)
    at check (/app/node_modules/fast-check/lib/esm/check/runner/Runner.js:62:11)
    at <anonymous> (/app/configuration/fc.setup.ts:197:14)
    at assertWithLogging (/app/configuration/fc.setup.ts:125:3)
    at <anonymous> (/app/repo/lib/pver_47c8185a-2747-4e5a-8384-01d7f81eb98d.test.ts:39:6)
Unit Tests 
// Unit Test for "Error Handling for Missing or Invalid Input": Handle cases where the 'files' array is empty or 'editInstruction' does not provide a valid string, to avoid malformed output.
function benchify_s(s) {
    return s.replace(/[^a-zA-Z0-9]/g, 'a');
}

it('benchify_s_exec_test_failing_0', () => {
  const args = superjson.parse(
    '{"json":[[[{"path":"(eN^%","content":"zaamah1"},{"path":")pi","content":"aWaaa4xaaYO"},{"path":"9 tZ2","content":"za"},{"path":"b=*wage[ #","content":"ur"},{"path":"A Ny*i/s","content":"NjaaUMaaK"}]]]}',
  );

  benchify_s(...args);
});

const filesContent = files.map(file =>
`### ${file.path}\n\`\`\`\n${file.content}\n\`\`\``
).join('\n\n');

return `Here are the current files:

${filesContent}

Edit instruction: ${editInstruction}

Please update the code according to this instruction and return all files with their updated content.`;
}

export const TEMPERATURE = 0.7;
export const MODEL = 'gpt-4o';