# "It runs, but it's wrong" — the quietest and most common AI-generated bug > The most common defect in AI-written code isn't a vulnerability — it's a result that is simply wrong while nothing errors. Totals that don't add up, search results that drop rows, failures reported as successes: how they happen, and how to find them without reading code. - Published: 2026-08-06 - Author: 友田 陽大 - Tags: バイブコーディング, AI駆動開発, 型安全, 個人開発, テスト - URL: https://tomodahinata.com/en/blog/ai-generated-code-silent-bugs - Category: Running the app you built with AI - Pillar guide: https://tomodahinata.com/en/blog/ai-app-pre-launch-guide-for-non-engineers ## Key points - The most frequent and serious defects in AI-generated code are error handling and business logic — more common than vulnerabilities, and invisible for months because nothing errors. - Five recurring patterns: swallowed exceptions, date boundaries, non-deterministic ordering in pagination, rounding drift, and ignored partial failures. All of them look normal on screen. - AI is excellent at the happy path and writes nothing for abnormal conditions unless told to — and abnormal conditions are precisely what production consists of. - The fix is to write down the conditions that must never break, in plain language, first — then have the AI turn those into tests. You can state conditions without reading code. - 'The AI said it fixed it' is not confirmation. Re-running the same check after the fix is the only verification available in this area. --- We've been talking about security, but in fact **the most common defect in AI-built apps is not a vulnerability**. **No error is raised, and the result is simply wrong.** That's it. What makes this kind of bug awkward is that there is almost no opportunity to discover it. The screen looks normal. Nothing errors. Nothing lands in the logs. A number is just slightly off. So it goes unnoticed for months — sometimes forever. --- ## 1. Why AI produces these AI is extremely good at writing the path where things go well. Ask it to "show a list of the user's orders" and it writes code that works perfectly for a user who has orders, with valid data, accessed by one person at a time. What it doesn't write is the cases below: - There are **zero** orders - The date sits **exactly on a boundary** (23:59:59 on the last day of the month) - **Two people act at once** on the same record - An external service is **slow or down** - A number is **negative** or absurdly large - A string contains **emoji or unusual characters** These are called abnormal conditions. And — **abnormal conditions are exactly what happens in production**. The reason is straightforward: they weren't in the instruction. AI isn't being careless; you didn't say "when there are zero orders, display 'no orders yet'." Of course you didn't. Only someone who has been burned by it once thinks of that. --- ## 2. Five recurring patterns Here are the shapes I actually see. All of them look normal on screen. ### Pattern 1: swallowed exceptions The most common, and the most dangerous. Code catches a failure and then **does nothing**. The error reaches neither the screen nor any log, and processing continues as though nothing happened. ``` [what you get] - "Saved!" when nothing was saved - a completion screen when no email was sent - an order confirmed when the payment failed ``` AI writes this because it is **trying to stop errors from halting execution**. Tell it "an error appears" and it will make the error stop appearing — and sometimes that takes the form of ignoring the error. **How to check**: make it fail on purpose. Cut the network, leave a required field empty, enter an absurdly long string. If it still says "success," something is being swallowed. ### Pattern 2: date boundaries In aggregation or filtering, the boundary day (or second) is off by one. ``` [what you get] - a monthly total omits the last day, or includes the first of the next month - "today's orders" misses orders placed around midnight - a timezone shift makes Japanese morning count as the previous day ``` The nine-hour gap between Japan time and UTC is a place AI gets wrong especially often. It looks right when you test it, and drifts only during late-night hours. **How to check**: create one record at the end of a month and one at the start, and confirm both land in the correct month. Try records around midnight too for extra confidence. ### Pattern 3: non-deterministic ordering in pagination If a list shows "20 at a time" and **the order isn't specified, entries drop or duplicate on page two**. A database considers itself free to return rows in any order when none is requested. Getting a different order on page one and page two genuinely happens. PostgreSQL's own documentation states that a `LIMIT` without `ORDER BY` yields an unpredictable row order. ``` [what you get] - paging through the list, some entries never appear - the same entry appears on two pages - fine while data is sparse, suddenly wrong once it grows ``` That it **only surfaces once data grows** makes this particularly nasty. **How to check**: enter about 30 records and compare pages one and two. Confirm the total is 30 with no duplicates. ### Pattern 4: rounding drift In money or percentage calculations, the handling below the decimal point slips. ``` [what you get] - the line-item sum and the displayed total differ by one unit - tax-inclusive → tax-exclusive → tax-inclusive doesn't round-trip - applying a discount makes the total go negative ``` Even a one-unit difference is a problem in accounting. The standard practice is to hold money as integers in the smallest unit rather than as decimals, but AI will use decimals unless told otherwise. **How to check**: try amounts that produce fractions, and **compare the total on screen against adding the line items yourself**. ### Pattern 5: ignored partial failures When several operations are performed together, a failure partway through doesn't stop the rest. ``` [what you get] - the order was created but stock wasn't decremented - the user was deleted but their data remains - the payment completed but no order record was created ``` You need a mechanism that guarantees "all succeed or all fail" (a transaction), and that too has to be asked for. **How to check**: hard to spot by eye. Use the instruction in the next section and have an AI investigate. --- ## 3. How to find them — what you can do without reading code There is one principle: > **Compute the right answer by another route and compare** Whether the number on screen is correct cannot be determined by looking at the screen. You need an independent check. ### Method 1: hand-calculate with very little data **This is the most reliable.** With only three to five records entered, compare the on-screen aggregate against your own arithmetic. With so few records you can check it in your head. Most people try to test by entering lots of realistic-looking data — but then you can't verify anything. **Deliberately keeping it small** is the trick. ### Method 2: aim at the boundaries Try the edges, not comfortable middle values. - **Zero** records - Exactly **one** record - **End of month / start of month**, **around midnight** - Amount **zero**, **negative** - A name that is **one character**, **only emoji**, **very long** AI doesn't get middle values wrong. It gets edges wrong. ### Method 3: make it fail on purpose - Close the browser mid-operation - Cut the connection and submit - Press the same button twice quickly - Do the same operation simultaneously in another tab "Press the same button twice" is especially effective and exposes duplicate registration or double submission immediately. --- ## 4. The fix — write down what must never break, first This is the heart of it. Most people ask an AI to "write tests." That's fine as far as it goes, but **AI writes tests against the implementation it produced**. If the implementation is wrong, the test encodes the same mistake. The test passes and the bug remains. What works is **reversing the order**. ### What you write You can't write code, but **you can state, in plain language, the conditions that must always hold**. In fact you're the only one who can — you're the one who knows what the app is supposed to do. Examples: ``` - an order's total always equals the sum of its line items - running the same search twice returns the same rows in the same order - paging through every page yields every record, with no duplicates or gaps - a balance is never negative - deleting a user leaves none of that user's data behind - a monthly total covers 00:00:00 on the first through 23:59:59 on the last (JST) - if payment fails, no order is created - if saving fails, an error is always shown on screen ``` Write these **first**, then ask the AI to "turn these into tests and check whether the implementation satisfies them." Where it doesn't, the implementation is what gets fixed. In that order, the AI cannot justify its own implementation. --- ## 5. Instructions to paste into your AI tool ### Investigate the current state ```text Read this app's entire codebase and list places that are problematic on the following axes, as a table. Do not fix anything yet. (1) Places where an exception is caught and nothing is done (swallowed) — where the failure reaches neither the user nor any record (2) Places where date/time handling looks doubtful, especially timezone conversion and whether month start/end are inclusive (3) List queries with no explicit ordering (ORDER BY), particularly paginated ones (4) Money or percentage calculations using decimals, or where the rounding direction is unspecified (5) Places that perform several updates together without a transaction (6) Places with no branch for zero or one records For each, explain in plain language under what conditions it goes wrong and how it would look wrong on screen. I cannot read code. ``` ### Turn your conditions into tests ```text Below are conditions that must always hold in this app. Write them as tests and check whether the implementation actually satisfies them. Where something is not satisfied, fix the implementation rather than the test. - (list the conditions you wrote out here) Tests must include boundary values: zero records, one record, end of month, midnight, amount zero and negative. When a test fails, explain in plain language which condition failed and which part of the implementation caused it. ``` --- ## 6. Don't believe "I fixed it" One last habit, the most important one in this area. AI reports "fixed." Usually it genuinely did. But **believing it without checking is especially dangerous for this class of bug** — since nothing errors, nothing happens even when the fix didn't land. After a fix, **check it the same way you found it**. - Compare against hand arithmetic with very little data - Try the boundaries (end of month, zero records, one record) - Make it fail on purpose Two or three minutes. Whether you have this habit is the difference between an app that is quietly broken after launch and one that isn't. --- ## Summary - The most common defect in AI-generated code isn't a vulnerability — it's a wrong result with no error - AI is excellent at the happy path and writes nothing for abnormal conditions unless told to. Production is abnormal conditions - Five recurring patterns: swallowed exceptions, date boundaries, non-deterministic pagination, rounding drift, ignored partial failures - To find them, compute the answer another way: little data plus hand arithmetic, aim at boundaries, make it fail on purpose - To prevent them, write the conditions that must never break in plain language first, and have those turned into tests. You can state conditions without reading code - "The AI said it fixed it" is not confirmation. Re-check the same way Security holes only cause harm if someone attacks; **this class of bug happens on its own, guaranteed**. In priority terms, the realistic order is to close [the four pre-launch items](/blog/ai-app-pre-launch-guide-for-non-engineers) and then come here.