Volumetric video! Nice!!

Imagine a next-gen VR experience that lets you speak realistic scenes, intelligent characters and complex situations into being, then interact with them in real time. It’s coming, due to a convergence of tech like this advance in real-time 3D video.

https://newatlas.com/vr/holodeck-real-time-volumetric-3d-video/

https://x.com/minchoi/status/1868137176254570810

Some videos in here that show the 3d point clouds that are generated.

I think in this blog for CWT I mention a possibility like this - viewers having the ability to transport into another character’s perspective, change setting, culture, etc.

A slight diversion, nigh but a short tangent, for a Spoke feature set marketed toward writers/creatives. The inverse of having AI bots create content is to have the AI bots enhance content. This opens up the path for the evolution of a new way for creatives to develop their vision. Also, why not hitch a ride on the AI hype train while its still hot? We can get in some RW marketing, Partner marketing, Spoke marketing and enhance the process of creative exploration all at once. It will be optional, of-course, for any specific post or project. It will not be a fit for every project.

Spoke AI could make project visions come alive.

Side Note: This could be a strong creative tool for anyone. If we create an amazing tool for creatives, but do not offer ways to keep certain ideas private, they will seek other tools to meet this need. We should respect the privacy of users and let them keep things private. Keep them on the platform for both open and private efforts.

There is a future, a very near future, where writing a journal entry, blog post, or similar will be very close to the concept of directing a movie. A future (I do take this to the eventual extreme) where a written post is immediately turned into an experience complete with generated sights, sounds and eventually sensations like touch, smells and feelings of the scene. All of this could be generated from embedded context. A future where the writer can direct all of this using the pen (or keyboard). This future will be amazing, and part of it is already here. Spoke can feasibly unlock the ability to enhance each post with AI generated content now.

“A director is one who makes the scenes of a film come to life… manages the artistic aspects… and coordinates with the cast and crew….”

A Cinematic Writing Tool gives writers the ability to be a pseudo-director in near real-time by bringing their vision to life immediately with AI. Sure, things may not align perfectly with what the writer imagined (cue the Texas Instruments AI Brainwave Interpreter Hat here). Sure, things may need to be tweaked. However, the potential for a story to instantly be brought forth to multiple mediums is very compelling. This opens a single story to many different audiences in one pass. Marketing, ideation, evaluation, creative debris, these things are all part of the creative process. Spoke AI can be a second, third or fourth pair of hands to help craft someone’s vision. The reality of most projects is that the end-result is often not the same as the vision we have in our heads. We still create. A CWT offers a new way to create. A faster product to market. More potential futures to sift through. More ideas to consider. A new way to breathe life into ideas.

Grand View

We are used to traditional books in our hands and movies on a flat screen with no intelligence behind it. These things are presented to us as-is and do not change. Ultimately, AI-backed entertainment has the potential to disrupt these mediums in ways we cannot imagine. The concept of loading a project into a database, letting AI render the scenes on-the-fly, and giving users the power to manipulate the scenes within the boundaries of the story-line opens up unheard-of freedoms for the viewer. How about changing a character’s appearance, or the environment, at will. How about moving a scene from the desert to a snowy landscape and keeping everything else the same. How about viewing everything from a different character’s point of view? The ability to change a character’s race, language, culture (smells!) on-the-fly is going to enable stories to connect across cultures like never before. These things all seem possible with AI, at some level.

Other Thoughts (messy draft format)

This tool dovetails nicely with Spoke’s current trajectory regarding project blogs. We could use this tool to scope out someone’s idea. This tooling could be used to create marketing materials for a project. It could be used to convert a project to a story format, with images, sounds, etc. The Cinematic Writing Tool would simply imagine a future, or multiple futures, where this is reality. What better ideation or marketing material development tool could one ask for?

Technical (not really) Implementation Ideas

Contextual Input Options:

• Notes (Hand Written, Photographed, Uploaded & AI Parsed)

• Spoke Posts (Overall Project Posts)

• Spoke Context Addons (Contextual addons to each project post)

• Images (Uploaded and AI descriptions extracted Mini-GPT

• Audio Recordings (Extract text from speech; audio context from background noise)

• User Prompts

All of these things could be used in conjunction with each other to create a scene. The user can pick and choose what is included as context. The user can tweak these things over time and regenerate things. The user can store copies of the generated content in a debris folder. The user can export the generated content for their own needs.

The pieces are already there. AI can generate sounds based on descriptions. AI can generate images (and video, but currently meh) from descriptions. AI can generate human realistic voice-overs. Figuring out how to mesh the results together may be the tricky part.

Allow the user to choose what contextual attributes are used. Three possible sources of context include the user may want to manually add context, or have Spoke pull context from the post, or the user may want Spoke to record audio around the user and store as context to play back.

Compression is an interesting option. There are examples of LLM’s compressing content, essentially creating a unique language, that it can then unpack and get very close to the original content. This would be an interesting way to store contextual data (eyeballs audio content) which could be ‘expanded’ later to generate audio for the scene. I am not sure of a way to extract and describe noises within an audio clip. But, this is along the lines of visual feature extraction so I am sure there is a solution.

Relevant Links:

Text to Audio: https://huggingface.co/spaces/haoheliu/audioldm-text-to-audio-generation

Image to Description: https://minigpt-4.github.io/

Export to HTML / SVG / MP4 / ETC?

Generated Output Ideas

• Youtube Shorts

• HTML Slideshows w/music (insert into blog?)

• Audio/Video montage

• Voiceover Narratives

• Sound-enhanced Storylines

• Project related ‘debris’

These would be configurable. Each project could have defaults.

Marketing Play

Partners???

• Jekyll is already integrated into Spoke.

• MAJR - This is similar to MAJR, although the inverse in that it creates sort of ‘debris’ of a person’s vision.

• Automattic - Day One could use our API to enhance their journal entries.

• RedwoodJS

• AI Hype Bandwagon (Splashy)

• Spoke Growth

• Spoke Feature Enhancement

Early Prompt Ideas

Summarize post and suggest image sequence in the style of _____ artist/director … and extract imagery descriptions in the style of ____ artist … and suggest score in the style of ___ and sounds in the style of _____ theme or movie

“One-shot prompting” - use example prompt to prime the LLM to give response in desired fashion. IE, if a project is sent to the generator: “Please suggest typical debris that a ____ type project might have. Use this list of debris for a web application as a template: “marketing materials, code snippets, application screenshots, customer reviews & testimonials, customer support, etc.”

Suggest length of montage, styles, etc.

Presets: Prompt presets for predefined patterns for music, score, etc.; Prompts based on user styles & previous themes

Rankings: Create a leaderboard with views/counts/votes/comments etc.

Dead Ends?

What constitutes a scene? How is the length of a scene, the transitions between visuals or auditory changes, etc. determined?

Spoke Feature Needs

Draft Versions

Private Versions

Subscriber-only Content

Billing system (to accept payment for generated content)

Better editor - I don’t want to have to edit my drafts in another program, just to copy over to Spoke. I do like other programs, but Spoke can provide a better editing experience.

Future Features

Local version, leveraging APIs or local ML Models, could be kept completely private.

Bookmarks

Gotta have this in there! https://vcai.mpi-inf.mpg.de/projects/DragGAN/

https://www.promptingguide.ai/introduction/tips

Sandbox

Writers are not used to having a tool that can bring their vision to life in near real-time.

On the other side of the coin, brain waves being interpreted by AI could lead to some amazing neural journaling tools!