System Art - Semantic, Interactive, and Modular

Important: Interaction should contain semantic information. Yet should be also simple & intuitive for first comers to use.

Multiple strategies to employ: 
Non-semantic interaction transformed into semantic interaction (Dimensional Amplifying/Transformation/Amplified Interaction)
OR Semantic Interaction make it phenomenological / Starts from semantic interaction but end up in non-semantic era

For example: Our daily interactions within platforms - Stars from semantic interaction but mostly subcognitive phenomenological interaction

Within the context of system art: Let’s say you’re controlling the system params / switching system context. There might be couple of buttons in front of you. First user clicks on those buttons for the sake of the signified meaning of the button. Over time, user clicks just for the sake of enjoying the output effect that the interaction causes, rather than attached to the meaning of the button.

This is crucial because most of users in interactive art does not want to confront with the heavy semantics, in most of the cases they are quite passive users - also related to reason why many of media art from now focused on the very simple action (waving hands, bodily movement) to create audio-visual affect for the interactive art, not nothing more - some kind of high-dimensional semantic interaction.

Although I think the semantical interaction (of user choosing smth semantical/input smth semantical) is very important, we cannot expect large vast majority of users to follow these semantic rules - For instance, limitations of text input/voice input

When given text input/voice input, it is largely observed that user doesn’t know what to say/input in an exhibition setting, and end up just entering dummy word. If it’s a word-based interaction, it’s still ok, but if it’s sentence-based interaction and users are enforced to input a large sentence, they most likely to leave - Especially when there’s not so strong background context given. So we should recall that this text-input interaction, especially when given at an early phrase of interaction, will lead to a disaster - interaction just for the interaction’s sake, not really gaining any useful information from the user side.

There is still a lot to be resolved with the voice input interaction - one of the hopeful hint is the openai’s realtime solution, enabling real-time voice interaction. But still, the same problem arises - user won’t have much to say from the very start.

Therefore, we kind of conclude that user is unable to give a bunch of context (in other words bits of information - if user gives plain text, that contains a lots of information, ideally. But in reality, they just end up giving dummy text, which contains no information) - we should narrow down the options that user can take (at least initially) and this is why the button-based interaction, option-based/parameter-changing interactions is more realistic option for us asking the user to interact which contains semantic interaction. (Or mapping the non-semantic interaction - bodily interaction into semantic dimension - dimensional transformation)

Buttons, Options, Sliders, Toggler, Checkbox, Multi-touch, Number Input, List Selection…  → Type A (Direct, Explicit Interaction)
Eye Tracking, Voice Recognition, Body Movement, Head Pose, EEG, … -> Type B (Implicit, Sensor-based interaction)

Text Input, Voice Recognition → Type C (Language based semantic interaction)
External APIs, External Data → Type D (Environmental/Contextual)

For long time it was considered that type b is more immersive/playful than type a – surely it is more advanced technology, but what i mean here by ‘Semantic interaction to phenomenological interaction’ is arguing that combination of type a (direct, explicit interaction) in a wisely designed & nudged way can also lead to very immersive, unique, playful, somehow dadaistic, and phenomenological interaction. (That being said, type b can be also, in other direction, lead to semantic interaction – more on the dimensional mapping/amplified interaction)

Generative Junk site example: Audiences, when interacting, gives new context - via buttons or list selection – this is normally very semantic interaction right? And let’s say there is a 대분류, 중분류, 소분류, and user slides to different 분류 columns and select one of the types – as they interact, first they just firmly ‘click’ on each button individually, but as the interaction progresses and accelerates, the start to ‘slide through’/within the buttons - Their finger, beforehand, was discrete movement (selecting a button and then leaving the interface touch). But now their finger does not detach from the interface - their finger is sliding through/touching on top of interface moving from a button to another, switching 대분류/중분류/소분류/context firmly, softly, phenomenological, somehow detached from the semantic of the button (or actually leveraging in an increasing speed) - and on the screen-side, junk sites are generated/altered in real-time matching with this context.

Related to multi-touch? How to develop this firmly & well-functioning on react?

“SLIDING THROUGH/ON TOP OF THE” Interface!
Ice skating on the interface - ice skating with your finger

This can be applied for other interaction types - sliders for example.

Multiple sliders columns - Like the iphone tab
Sliders - metrics - parameters - all adjusting the current
Mastering the sliders / numerical value adjustment (param adjustment) is the key to designing semantic-to-phenomenological interaction

Sliders displayed in column - arranged in row will have very different experience from sliders displayed in row - arranged in column. Why couldn’t I think of the sliders displayed in the column? Sliders displayed in columns are way much more of a playful experience - We are used to scroll vertically rather than horizontally.

Also use page structure wisely: Scrolling up & down! Scrolling is really a powerful tool, and it is the most representative interaction that we do nowadays in the contemporary world. What if scrolling up & down is combined with navigating back & forth - the most universal layout used in contemporary sns/platforms/websites - but make it more dadaistic! Brutal! Focusing on the core interaction!

Page Group A (대분류)
Page Group B (중분류)
Page Group C (소분류)

A ←> B ←> C: Horizontally connected,  back & forth, (Click and go forth, go back - left movement and go back)
Within A/B/C: Vertical interaction, scroll interaction

Hierarchical interaction & Semantic & Phenomenological

More advanced usage?

Hacking Shopping Mall UI: You shop through different contexts! (Trump shopping greenland, Panama, Canada Amazon UI)
Hacking Flight Reservation/Hotel Resera\vation UI (All sort of shopping mall - kayak, booking,com)

Map → Good Example of Semantic Interaction (Interaction w/ Objective) & Phenomenological Interaction (Navigating, Swirling around) combined → The Zoom & Pinch is quite sense-based but also contains semantic meaning. It gives you an experience of ‘finger skating’ on top of the interface

Punctum Interaction: 위태로운 시스템, 벼랑 끝의 사회… 겨우 평형을 유지해가고 있는 시스템
시뮬라크럼! 시스템 자체가 거대한 시뮬라시옹, 약간의 푼크툼 (외부 충격 - 인터랙션 - Interaction at Mass speed)으로 사회의 외상이 드러난다… 그러나 이후 시간이 지나면 다시 평형 상태로 돌아가는 (푼크툼을 애써 지워내는)... 여기서 푼크툼-외부 충격은 현상학적 인터랙션의 형태를 띈다! (오메가의 예시도 약간 유사함)

Equilibrium? System? External Shock?

Technically: Reference Hammer.js!

—

o1-pro summary

Interaction as a Spectrum:
Users start with semantic intent (e.g., pressing a button for its meaning) but transition into fluid, subcognitive engagement (e.g., sliding through UI elements, focusing on experience rather than meaning).
This transformation mirrors modern digital behaviors (e.g., platform engagement, UI-based habit loops, scrolling culture).
UI/UX as a Medium for System Art:
Experimenting with button-based, slider-based, and list-based interactions to induce flow states similar to gaming or shopping interfaces.
Leveraging multi-touch, scrolling, hierarchical navigation (A → B → C structures) to nudge users into deeper, emergent interactions.
Conceptual Parallels to System Art:
The system itself is self-sustaining, semi-automated, and partially obscured—users see surface symptoms but not the core dynamics.
The interactional equilibrium can be disturbed by external shocks (Punctum Interaction) but eventually self-regulates, referencing cybernetic systems, societal feedback loops, and hyperobjects.