Whiteboard Snapper
Whiteboard + Projector Two-way Interaction - 2023-08-12
- Provide a TLDraw surface for the visiting user to draw on.
- Project the information from the TLDraw onto the whiteboard
- Grab the drawings on the whiteboard
- Process and clean them up
- Omit the visiting user's drawings (subtraction?)
- Add the drawings on whiteboard as a video feed element in the visiting user's TLDraw
- Make the element non-editable, so the user can draw over the top of it
- Save the state of the whiteboard by tapping a "button" on the whiteboard itself
- Button can be actuated by the camera watching for a complete April tag
- Write the app as a CLI-powered TypeScript app using ECHO
- that allows me to quickly get the whiteboard photos into my knowledge management system and integrate them into my work
Process
- Take a picture using ffmpeg.
ffmpeg -f avfoundation -framerate 5 -video_size 3840x2160 -i "1" -vsync 2 -vframes 1 test3840.jpg
- Use Fred's whiteboard script
~/Downloads/whiteboard -c "262,241,2822,223,2661,1982,571,2028" -a 1.25 -t 40 ~/whiteboard.jpg ~/whiteboard-smoothed.jpg
~/Downloads/whiteboard -c "422,263,2957,227,2780,2015,665,2000" -a 1.25 -t 40 ~/test3841.jpg ~/whiteboard4-smoothed.jpg
~/Downloads/whiteboard -c "418,74,3419,0,3222,2129,735,2122" -a 1.25 -t 40 ~/test3841.jpg ~/whiteboard4-smoothed.jpg
~/Downloads/whiteboard -c "462,182,3293,157,3092,2146,752,2128" -a 1.25 -t 40 ~/test3841.jpg ~/whiteboard4-smoothed.jpg
sed fails unless I export some random shiz:
export LC_CTYPE=C
export LANG=C
Explanation here.
2020-09-07
Progress! Managed to get my Brio to capture a 4k (4096x2160) image of my whiteboard!
Here's a good shot of why that matters ^
4K resolution captures a lot more detail with a lot more fidelity, which gives a lot more information to the imagemagick script to process.
- Tried VLC, tried ffmpeg
- Turns out VLC was using ffmpeg under the hood
- Got ffmpeg working using the AVFoundation framework
- I’ll need a script that wraps ffmpeg to find the index of the Brio, as it will change
ffmpeg -f avfoundation -framerate 5 -video_size 3840x2160 -i "1" -vsync 2 -vframes 1 test3840.jpg
- Test if I can use different cables and still achieve the higher resolution
Whiteboard Workflow CLI Tool
Build a custom workflow that would allow me to trigger whiteboard capture from anywhere and do a variety of things:
- Copy the picture to my clipboard
- Upload the picture somewhere semi-public with a URL that I can share with others
- Copy to a whiteboards directory locally with timestamp so I can page back through my whiteboards
- Print the whiteboard to the printer in my office
Building the Whiteboard CLI tool in Rust
- Mac requires interaction with webcams via AV Foundation, which has to be interacted with via Objective-C. There is a Rust - ObjC bridge that could be used to access AV Foundation directly.
- Another option would be to just wrap the command line tool FFMPEG, which is how I'm currently accessing it. Here's an example of wrapping FFMPEG in Rust.
- A third option would be to rely on OpenCV and use the Rust-OpenCV wrapper to take the screenshot.
Cleanest and easiest option would be to use Rust's ObjC bridge and just grab the screenshot directly. This would allow me maximum control over the code, while having a very minimal library with zero dependencies. The one sad part is this would only work on Mac. We could do a platform detection and someone could extend the library to work on Linux.
Prior Art
Clearboard by Hiroshi Ishii and Minoru Kobayashi