The Future of Social Media in 3D+
So the announcement of the forthcoming 3D posting capabilities on Facebook has me terribly excited, but also incredibly frustrated over the limited scope that they and basically everyone else in ALL of the cross-relational industries are handling the transition to 3D. These systems need to handle the real, 3D data collected from sensors–3D models of real world objects and point clouds of locations — not just the low hanging fruit of 3D modelling for the gaming, special effects, and animation communities.
Last year, I had the interesting and disorganized pleasure of being recruited by Facebook, presumably to chat about similar topics. In the end I opted to accept the pushier, more organized recruiting efforts of global design firm AECOM to come play on the East Coast in the government arena (of which, I cannot write without like 8 levels of editors and security checks, so anything fun on that will have to wait).
BUT since Facebook seems to be following this chain of thought, but is a teensy bit behind (and as previously mentioned, incredibly disorganized about their recruitment of scientists)–I thought I’d offer up to the denizens of the internet the quick brainstorm I wrote up for the Facebook recruiter amongst our assorted chats. I doubt the email chain has made its way to anyone useful, so here’s hoping some of the below makes its way somewhere and sparks some wider levels of interdisciplinary dialogue.
Bear in mind, I’m not gussying this up, just pulling straight from one of the overview chats with the Facebook contact, and though there were NDAs in place on things I saw at the Facebook HQ/events, none of those pertained to the actual recruiting dialogue–at least not at whatever starting level this was at — so what follows was more of a rough, off the cuff break down of what I would want to work on if given the research lab at Facebook that was on the table:
…I work with imaging tech and visualization systems for collecting and sharing 2D and 3D information about space, time, and the people that inhabit it. I use technology to find better ways to image and share the past and present in order to build better visualization and connection systems for the future. I work along that whole data pipeline — evaluating new tech, helping develop the data collection, data use, and tech further for expanded use cases, and anticipating and reacting to cultural and policy shifts in relation to its development and use.
Here’s what that would mean for something like Facebook/Instagram/Oculus/etc:
*Using Real Data in VR/AR/MR
— At the super basic end of what I do is push for all of the scientific 2D and 3D data all around us to be used in new media systems like virtual reality — especially when it comes to cultural heritage sites like famous monuments and museums. They’ve been some of the best test grounds for using all the fun and fancy sensors and gadgets and gizmos to digitize time and space and objects therein — it’s perfect content fodder and a great place to experiment in (especially in terms of National Geographic public relations re: conservation, preservation, and all that jazz). But that’s just the tip of the iceberg…
*Moving Towards Ubiquitous Sensing
— Society is moving beyond 2D image collection into having the power to collect constant barrages of 3D data — that means there needs to be an infrastructure in place for all of that 3D information to go — a place for all of those real world models collected by LiDAR, photogrammetry and other data collection techniques currently only really in use by certain industries (like my own). As everyone gains access to these systems via the forecasted smaller sensors in our smartphones, autonomous cars, and other sensing systems, they’re going to need a place to share them and link them to the rest of their lives. We’ll need a place not just for our 2D selfies, but our 3D ones from MRI and CT scans and photogrammetric models. Instead of just uploading an album of photos from a recent trip, we need to be able to upload the 3D models we made of the beach in Hawaii and have ways to share them and continue exploring them — via traditional 2D systems (desktop, phone, etc); the emerging virtual, augmented, and mixed reality systems; and who-knows-whats-next-but-its-going-to-be-exciting-multidimensional-visualization-systems-beyond-those.
That means adjusting file format capabilities and engine power beyond the base processing of jpeg, gif, standard and 360 videos etc, to start growing mechanisms to share multidimensional data (and encouraging its ubiquitous content creation in the process — thus spawning reams and reams of data of geographical, cultural heritage, medical, and all sorts of other categories in the process). That means prepping Facebook’s VR and AR and everything else for carrying real actual data collected by a sensor vs created by an animator. All of that lovely data — all those photographs and models etc — easily translate into accidental and potentially awesome citizen science programs of mass engagement and awareness of global concerns and culture.
But more importantly, it would also translate down the line into a constantly updating, ever growing digital scaffold of the world, complementary to remote sensing tools, that serves not only as the ultimate base for virtual reality systems as a real world holodeck style deal, but as mega crucial data that government and commercial groups the world over have expressed interest in for analytics. In historical terms — such a digital scaffold is also the ultimate record of time and space, generated by its users — allowing future generations to hop back in time via Facebook and experience an event in unprecedented ways. (All of which, needs to appropriately secure and allow for ethical levels of access.)
*Redefining Data Collection & Cross-referencing Mechanisms
— Expanding what data is possible to share -will change what data is collected. It would redefine ‘data’ and open the floodgates towards everyone collating not just their food or travel experiences, but contributing to wider levels of observational science and record-keeping.
Facebook is an archive of the human existence. And right now, it doesn’t have the capacity to link up all the different multimedia facets of life — but it easily could by expanding in further phenomenological directions of data capture and sharing. Being able to add in more types of data with additional layers of metadata and paradata to photos and posts adds a richness to the system as we know it. If you can create an album of your Hawaii trip that isn’t just photos embedded in a 3D model but also sounds and notes and videos and maps and cross-referenced menus of restaurants you went to etc (and which could theoretically pop up at the other end, so friends in Hawaii could access it there and walk past a place and have it ping at them letting them know you’d been there and linking you to their multimedia extravaganza) — it’s a game-changer of experiencing the world and accessing cumulative knowledge bases. Being able to start collecting and collating albums of things beyond photos and posts adds a whole new level to what Facebook is capable of. And it needn’t rely on the user to even create content in some instances — like if you’re walking through an art gallery or store, liking physical things along the way, and Facebook builds you an album/wish list/cabinet of curiosities of your trip via venue-created pictures representing the things you liked there — it would be a major anthropological shift. Having a mechanism that organizes our observational existence for us (as Facebook and Instagram have already started to do) revolutionizes the way we see ourselves and share ourselves with society and gives us a greater knowledge base to start learning and building more from.
— The cross-referencing possibilities for current data collected via Facebook et al and anticipated future multi-dimensional data from Facebook has boundless possibilities for cross-referencing and linking (thereby further enhancing its analytical powers). In addition to the potential for merely training image recognition systems, there are a myriad of ways to use comparative data to expand the daily use of and knowledge gathering from Facebook For instance, cross-referencing art and architecture into the mix. If someone likes X, Y, and Z and painting 214 has been tagged with those attributes X,Y, & Z — why not bring it to a user’s attention. If a user is visiting a museum and takes a selfie with or builds a rapid structure from motion 3D model of something in the museum — why not let them bring it to the museum’s attention (and temporal archives tracking artifact health). In future augmented reality systems, posts could easily connect a user to additional information about what they’re seeing (i.e. the most prevalent concept of the AR future out there — expanded annotation systems of everything from architectural features and shopping to history and plant identification). Or at its most simplest, sooner-rather-than later stage: If my grandmother and I are both on Facebook and she’s posted pictures of herself in post-World War II Paris, and I go to Paris now — Facebook could and should pull up her picture and prompt me to visit the same spot and collect a similar image, which could then be shared side by side.
*Scaling Connectivity Into The Past And Forwards Into The Future
Facebook is perhaps the greatest archive of individual human existence thus far. To ensure that it stays the greatest mechanism for future generations, it needs to ensure that it’s capable of looking backwards and forwards to build itself. Facebook needs to allow special mechanisms and encourage old albums of pre-digital photos to make their way online — thereby stretching out its content pre-2004 and pulling in the older generations as an ever-widening audience base to fill in their content blanks (and make increasing connections between old friends and family networks in the process). Cross-referencing historical archives and genealogical archives into Facebook’s systems would also mean pushing things out even farther and aggregating humanity’s knowledge about itself in a single, constantly growing spot. With all of the older data in place, suddenly it’s possible to not just link together family webs and show incoming generations the faces of those who have come before. It’s possible to start putting that into the context of their everyday lives. To not just show them timelapses of their own selfies aging throughout the years (which should be a thing), but to say, ah, you’d like to compare all the three years olds in your family to your current three year old, cool –here’s your great-great grandfather at age 3, and your favorite aunt at age 3, and your partner’s mom at age 3, and you at age 3, and your own three year old, etc all in a handy dandy little bit of media. Actually, I’d particularly love the timelapse feature to peruse Halloween costumes of myself and family members past and present. Pulling in older content would not only increase current levels of engagement with Facebook, it ensures that future generations would also be engaged, because Facebook is now the hub of their family lore. And in so doing, it creates a multigenerational database the likes of which the world has never seen — allowing for all kinds of new scientific and anthropological analytics into humanity.
Adding the past in alongside new mechanisms for collecting and sharing the present also ensures that Facebook would be an ongoing, thriving, ever-growing mega-content machine — with an expanded and ever expanding audience. As the ultimate resource on humanity, Facebook would never lose its place in human society. No one is going to give up on the social media mechanism that contains and connects them to their own personal past while simultaneously ensures that future generations in the far distant future will remember them more clearly than our current conception of individuals in history…..
The above isn’t a full summation of things, I’ve redacted some of the bits closer to what I’m still working on now — and a lot of things aren’t even covered in this bit of recruit-y brainstorm chat. BUT here’s hoping some of the above spirals a bit a bit further.
No one person or company or group is going to sort out the 3D data sphere–it needs to be an active, open collaboration across silos and across industries. So let’s get chatting already….
So let’s all dream big, and build a better AR Cloud, Social Media sharing system, 3D enabled business practices, and improved geospatial data tracking EVERYTHING and then some.
