René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Floating point / HDR texture support","articleBody":"Processing is tightly coupled to ARGB 8-bit textures via the `PImage` class hierarchy. When the project started, the only crossplatform drawing lib available in the JDK was AWT/Java2D. Those APIs expose pixels as int in ARGB ([`TYPE_INT_ARGB`/`TYPE_INT_RGB`](https://docs.oracle.com/javase/8/docs/api/java/awt/image/BufferedImage.html#getRGB-int-int-)), and [`java.awt.Color`](https://docs.oracle.com/javase/8/docs/api/java/awt/Color.html) wraps 0–255 channels. That layout provides zero copy access, blits via `MemoryImageSource`, and compatibility with other AWT stuff that's used pervasively throughout the codebase.\n \nEvery part of Processing assumes pixel data is stored as 32‑bit ints with 8‑bit channels. That assumption is baked into our public API (the \"pixels\" array) and is all over the place inside `PGraphics`, image filters, tessellators, OpenGL and future WebGPU backends, and font/shape caches, etc. Because there is no abstraction around \"pixel format\" or \"color storage\", any attempt to add floating point textures would require touching practically every file, amounting in an effective soft-fork.\n\n## Deep coupling\n\nHere are some more specific issues:\n\n- `int[] pixels` is exposed as public state and could be used from user sketches all the way down to renderer internals. Every consumer assumes 4 bytes per pixel with the same layout. Public fields are particularly problematic in Java as they can't really be overridden, just shadowed. \nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/core/PImage.java#L92\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/core/PApplet.java#L222\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/PGraphicsOpenGL.java#L5379-L5386\n\n- Color math is hardcoded to obtuse bit manipulation and 0–255 ranges eg `colorCalc` packs/unpacks ints, `PShape` duplicates this, and image blending/filtering is doing lots of `\u003e\u003e\u003e 24` \u0026 masks.\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/core/PGraphics.java#L7631-L7638\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/core/PGraphics.java#L7680-L7735\nhttps://github.com/processing/processing4/blob/main/core/src/processing/core/PShape.java#L3514\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/core/PImage.java#L2471-L2480\n\n- GPU texture uploads/downloads are always `int[]` → `IntBuffer` → `GL_RGBA`/`GL_UNSIGNED_BYTE`. See `Texture.set`/`setNative`, the `texSubImage2D(...PGL.RGBA, PGL.UNSIGNED_BYTE...)` calls, the `rgbaPixels` conversions that only handle ARGB/RGB/ALPHA ints and the `updatePixelBuffer` `IntBuffer` path.\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/Texture.java#L303-L305\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/Texture.java#L346-L347\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/Texture.java#L1000-L1061\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/Texture.java#L792-L795\n\n- Moving data between CPU pixels and GPU buffers is just \"reinterpret int as native RGBA\". `readPixels()` reads `RGBA`/`UNSIGNED_BYTE` and immediately calls `PGL.nativeToJavaARGB`. The reverse path (`drawPixels`) calls `PGL.javaToNativeARGB` before writing to the FBO.\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/PGraphicsOpenGL.java#L5389-L5411\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/PGL.java#L1646-L1653\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/PGraphicsOpenGL.java#L5437\n\n- GL resources themselves are fixed to 8‑bit RGBA, renderbuffers/textures are always allocated as `RGBA8`, and vertex colors/materials are uploaded as `UNSIGNED_BYTE` attributes.\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/FrameBuffer.java#L392\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/PGL.java#L2854\nhttps://github.com/processing/processing4/blob/11f1a1e666c7cec6ddf483c5c8fffb5b9009a137/core/src/processing/opengl/PShapeOpenGL.java#L5517-L5520\n\n- Even our new WebGPU backend just forwards the existing `fillR`/`fillG`/`fillB`/`fillA` values that originate from the same 8‑bit `colorCalc` method flow, so it inherits every limitation and would otherwise have to reimplement basically all of the existing API.\nhttps://github.com/processing/processing4/blob/1ee012e7942f0cfa9029151c6d7a1b992c044d0c/core/src/processing/webgpu/PGraphicsWebGPU.java#L56\n\n## Why is this an issue / What this limits\n\nAt a high level, while 8-bit textures may have been the norm in the early 2000s, the assumption has basically flipped. Modern GPUs execute most shading math in 32-bit lanes regardless of the source format, so even when sampling from an 8-bit texture the ALUs are still getting their parallelism over 32-bit registers. Unless you're explicitly using packed instructions (which is rare for generalist graphics work), the only real thing sthat 8-bit gets is memory bandwidth, which mostly only matters on low-end mobile. Everywhere else the industry is moving toward mixed precision (e.g. FP16 tensor cores for ML matmul ops) because compute cost is no longer tied to per-channel bit depth.\n\nHowever, specifically relevant to art, the lack of floating point textures is severe limitation to the following techniques.\n\n### Installation art\n\nStuff like motion tracking, heatmap accumulation, or any \"paint with time\" piece depends on adding and accumulating tiny deltas every frame. In 8-bit those changes round away, so small movement never shows up and trails die instantly. Float buffers let artists integrate small movement over minutes or hours and only quantize when sending to the projector. Multi-projector setups also have problems, feathering overlaps, gamma correcting different units, and warping/projection mapping content through multiple correction stages all require smooth ramps. Quantizing each stage to 256 buckets produces visible steps, which is especially problematic in dark rooms typical for these kinds of installs. Hardware sensorb ased work (think depth cameras, environmental sensors, etc) typically rely on filters and other post-processing to be useful for producing a final render for the projector.\n\n### Generative art\n\nFeedback is super common in generative art, ie read the last frame, do some texture processing, write it back, repeat. In 8-bit every read-modify-write cycle loses precision, so the loop collapses after a few iterations and colors wander off due to rounding and look bad if they work at all. Float textures keep temporal buffers alive and stable which is why applications like TouchDesigner or vvvv typically default to them. Popular algos like reaction-diffusion, fluid sims, and cellular automata also add/subtract tiny gradients every step. Quantization steps in 8-bit makes patterns fall apart. Floats match the reference equations. Even simple particles need floats to ensure the motion looks and particle sim parameters work from frame to frame. Layering hundreds of low-alpha sprites in 8-bit will jump between \"invisible\" and \"too strong\" because the intermediate alpha values simply don't exist. For people who care about color grading or LUT work, esp to reference stuff like mockups made by designers which is important in professional settings working on a team, you need headroom. Doing the math in 8-bit crushes blacks and create midtone bands which will make your designers cry.\n\n### Modern 3D graphics techniques\n\nIn the more game oriented world, TAA and other AA techniques assume you can accumulate subpixel jitter over many frames without the values collapsing. When you use 8-bit components, the first couple of blends push dark tones to zero and you're left with banding or ghost trails. Modern engines accumulate in 16/32-bit floats and only clamp when tonemapping out to the display. The same is true for HDR stuff like bloom, tone mapping, or PBR shading that need to push energy past 1.0, do postprocessing, then compress the result. With 0-255 you just clip to white and the math breaks. Deferred renderers and screenspace effect passes also suffer when encode normals, roughness, or depth in 8-bit and SSAO/SSR immediately reveal the quantization steps. Backends like Metal/Vulkan assume you can request `RGBA16F`, `R11G11B10F`, depth-only attachments, etc. which the user may want to display / debug. There's interesting Processing style techniques here that simply aren't possible atm.\n\n## Conclusion\n\nChanging Processing's texture format and color math is a breaking change at basically all levels of the existing codebase but is crucially important for modern graphics techniques.","author":{"url":"https://github.com/tychedelia","@type":"Person","name":"tychedelia"},"datePublished":"2025-11-13T23:07:25.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/1324/processing4/issues/1324"}