Meta SAM 3 Released: Video Tracking, 3D & Open Weights!

Meta's Segment Anything Model v3 Unveiled [00:00]

• Meta has released the third iteration of its Segment Anything Model (SAM).
• This model can detect and segment objects from images and videos.
• It can also track these segmented objects using simple text prompts.
• The release builds upon two previous iterations, indicating continuous development.
• The model is an "open weight" model, similar to predictive models, rather than purely generative ones like LLMs.
• SAM v3 has significant practical applications, particularly in data set creation for computer vision.
• Creating computer vision data sets can be expensive and time-consuming.
• The model allows users to select an item or object in an image, and it will expand that segmentation throughout the video sequence, detecting similar objects.
• Meta continues to release open-weight models, which is noted as positive, especially after the experience with LLaMA.
• SAM is described as a unified model for detection, segmentation, and tracking of objects using text exemplars and visual prompts.
• The weights are publicly available on Hugging Face, and multiple platforms allow for testing.

"These models have very important practical applications especially when it comes to data set creations because computer vision based data set can be extremely expensive and time consuming to create."

Technical Details and 3D Capabilities of SAM [01:34]

• The model functions with an image encoder and a text encoder for object detection.
• For tracking objects in videos, it utilizes a "memory bank" to keep track of segmented objects and their locations.
• Trackers are then used to update object locations, enabling simultaneous tracking of multiple objects.
• Meta also released a SAM 3D model that can take image or video input, detect and segment objects, and then create 3D models.
• This 3D capability opens up many new possibilities.
• The SAM 3D model is also open-weight and can be used in custom applications.
• Image segmentation was previously a difficult problem, requiring dedicated models like U-Net.
• Now, a single model can generate segmentation masks and create 3D models.
• Meta is open-sourcing a video data set that will be publicly available to encourage further development.
• The model is state-of-the-art for tasks like concept segmentation, visual segmentation, and object counting.

"Well, I'm glad that Meta is still releasing open weight models after what happened with the llama form."

Interacting with SAM: Playground and Video Cutouts [03:10]

• Users can try out the model through a new interactive "Segment Anything Playground."
• The playground offers various demos.
• Users upload a file, and the model detects different objects.
• Users can select an object, apply different effects, and share the results.
• The video demonstrates "video cutouts" using an 8-second video of a soccer game.
• To track an object, the process starts at the first frame.
• The model can detect objects in the current frame and then search for that object in the entire video sequence.
• It is capable of detecting the same object throughout the whole video.
• Objects can also be selected by clicking on them.
• There are instances where the model might have trouble tracking an object if it changes appearance significantly or if another similar object is mistaken for it.
• The model can segment multiple instances of the same object class, such as all zebras in an image.
• When uploading a personal thumbnail, the model detected items like a chalkboard, chalk eraser, hoodie, and a man.
• Effects can be applied to segmented objects, potentially for social media sharing.

"And now you can see that it's detecting the same object in the whole video, which is pretty awesome."

3D Scene Generation and Applications [06:21]

• A notable feature is the "create 3D scenes" functionality.
• Users can click on different objects in a scene, and the model segments them.
• Clicking "generate 3D" creates a 3D model for the selected objects.
• The model can create 3D boundary bodies, such as generating a 3D pose of a person in the same pose as the original.
• These capabilities suggest many interesting applications.
• Other example demos include generating bounding boxes around people and creating simple masks.
• These mask functionalities are useful for computer vision data sets.
• The ability to select one object and have it expand to similar objects across a video sequence saves time in data set creation.
• The model can track vehicles in a scene and even track license plate numbers.
• It can create motion trails for tracked objects.
• A "blur faces" demo allows users to track human faces and apply effects like blurring.
• These applications are possible because SAM v3 offers more control than previous iterations, allowing for both text and visual prompts.

"So, for example, here are different effects that are present. Right now, these are going to be pretty great if you want to share them on social media."

Developing with SAM: Notebook and Local Deployment [08:42]

• The video shares a notebook from Roamflow for building on top of the SAM model.
• The models are available on Hugging Face.
• Users need a Hugging Face token and must accept gated terms and conditions from Meta.
• A high-capacity GPU is required, such as an A100; a T4 GPU is not sufficient.
• The process involves installing required packages and loading the model, which downloads weights.
• An example video is provided to the model, and frames are extracted.
• Text prompts are necessary for the model to work.
• In the example, the prompt "track jets" is used on a frame from a video.
• The model visualizes the mask and counts five jets in the scene.
• These masks are then applied to the entire video.
• The model demonstrates effective segmentation and tracking even when a jet goes out of frame and returns, or when there is occlusion.
• The functionality will also be available in Instagram's video creation app.
• The model can be easily integrated into existing workflows.
• Meta's continued open-sourcing of these models is highlighted positively.

"This is going to be also available in the edits app from Instagram video creation app. I think it opens up a lot of different possibilities."