René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[Testing] Add Informal Test Suite","articleBody":"With the speed of iteration, it seems like the right time to introduce a very basic test suite to ensure there's no regression in accuracy. \r\n\r\nToday: We have 8 queries that we run against every PR to get a _vibe-check_ if stuff is breaking or accuracy has regressed. \r\n```\r\n- What are the three highest income zipcodes in {City}\r\n- Give me 5 zipcodes in {State} with the highest income and hispanic population of at least 2,000\r\n- Which 3 zipcodes in {City} have the highest female to male ratio?\r\n- Which zipcode in {City} has the most racial diversity and what is the racial distribution?\r\n- 10 highest crime cities in {State}\r\n- Which 20 zipcodes in {State} have median income that's closes to the national median income?\r\n- Which zipcode has a median income that is closest to the national median income?\r\n- What is the distribution of all crime in {City}\r\n```\r\n\r\nVibe-check: \r\n- Does the output make sense?\r\n- Is there a degradation in speed? i.e. is it fast (~2s for most e2e requests, sometimes 4s or 5s when the API has to retry)\r\n\r\nFor the API, we could make these unit tests (heads up - they could be very flaky because LLMs are non-deterministic). Verifying if the generated SQL matches the expected output may not work, but checking the table values might work! We could also just have the table values outputted and compare manually. \r\n\r\nFor the front-end, can we do something simple here? If not, we can just manually try the 8 queries in the Vercel preview deployments, that's pretty clutch already","author":{"url":"https://github.com/asaprahul","@type":"Person","name":"asaprahul"},"datePublished":"2023-03-13T22:31:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/82/textSQL/issues/82"}