Monday, September 9, 2019

Know Where Your Data Lake Has Been?



The foundation of a good data management strategy is based on number skills including data policies, best practices, and technology/tools as shown in the diagram above. The most commonly discussed term of the bunch, from the ones listed in the diagram, is Data Governance. This term is tossed around a lot when describing the organizational best practices and technical processes needed to have a sound data practice. Data governance can often be ambiguous and all encompassing, and in many cases what exists in  organizations falls short of what is needed in our modern big data and data lake oriented world where ever increasing volumes of data are playing an ever more critical role in everything a business does.

What is often left out and is missing in many modern data lake and data warehousing solutions are the two lesser known cornerstones I show in the diagram:  Data Lineage and Data Provenance. And without these additional pieces your data lakes (with ever increasing volumes and variety of data) can quickly become an unmanageable data dumping ground.

Who needs Data Lineage and Data Provenance management tools, APIs and visualization services?
  • Data Engineers (building and managing data pipelines)
  • Data Scientists(discovering data & understanding relationships)
  • Business/System Analysts (data stewardship)
  • Data Lake / BI Executives (bird's eye view of health and sources/destinations)
  • Data Ops (managing scale and infrastructure)
  • Workflow/ETL Process Operators (monitoring & troubleshooting)
Most of the ETL processing and dataflow orchestration tools out there in the market (open source and commercial) such as NiFi, Airflow, Informatica, and Talend among others,  do not directly address this gap. What is the gap? The gap is knowing where your data is coming from, where it has been and where it is going. Put another way, having visibility into the journey your data takes through your data lake and overall data fabric. And doing this in a lightweight fashion with out a lot of complex and expensive commercial tools.

Let's spend a bit of time talking about data linage and data provenance in particular and why they are important parts of a modern and healthy overall data architecture. First let's touch on the broader Data Governance ecosystem.

Data Governance

Data Governance can be an over used term in the industry and sometimes is all encompassing when describing the data strategy, best practices and services in an organization. You can read a lot of differing definitions of what data governance is and is not. I take a simple and broader definition for data governance which includes:
  1. Data Stewardship - org policies for access/permissions/rights
  2. Data Map - location of data
  3. MDM - identifying and curating key entities
  4. Common Definitions - tagging and common terminology
  5. Taxonomy/Ontology - the relationships between data elements 
  6. Data Quality - accuracy of data
  7. Compliance - HIPAA, GDPR, PCI DSS...
These are all important concepts, yet the do not address the dynamic nature of data and the ever more complex journey data is taking through modern data lakes and the relationships between data models residing in your data lake. This relates to needing to know where is your data going and where did it come from. This is where data lineage and data provenance comes into play to compliment data governance and allow data engineers and analysts to wrangle and track the run-time dynamics of data as it moves through your systems and gets combined and transformed on its journey and its many destinations.

 

Data Control Plane == Data Lineage and Data Provenance

I view data lineage and data provenance as two sides of the same coin. You can't do one well without the other. Just like digital networks have control planes for visualizing data traffic, our data lakes need a Data Control Plane. And one that is independent of whatever ETL technology stack you are using.

In our modern technical age, data volumes are ever increasing and data is being mashed and integrated together from all corners of an organization. Managing this from both a business level and technical level is a challenge. A lot of ETL tools exist today to allow you to model your data transformation processes and build data lakes and data warehouses, but they all consistently fall short of giving you a comprehensive and I would say orthogonal and independent view of how your data is interconnected; where your data came from, where it is right now and where it is going to go (and where it is expected to go next) in its journey through your data fabric and destinations.

Many modern ETL tools let you build visual models and provide some degree of monitoring and tracking, but these tools are proprietary and can't be separated from the ETL tools themselves which creates lock-in and does not allow one to mix and match best of breed ETL tools (Airflow, aws step functions, lambda, Glue, Spark, EMR...etc) that are now prevalent in the cloud. If you are using multiple tools and cloud solutions it gets ever more complicated to have a holistic view of your data and its journey through your platform.

This is why I strongly believe that data lineage and data provenance should be completely independent from what underlying ETL tooling and data processing technology you are using. And if it is not, then you are both locking yourself in unnecessary and greatly limiting the potential of your data ops teams, data engineers and limiting your overall management of the data and the processes carrying your data through its journey in your data lake.

Data Provenance and Data Lineage are not just fancy words; they are a framework and tool set for managing your data and having a historical audit trail, reat-time tracing/logging and control plane graph of where your data is going and how it is interconnected.

So do not build your data lake without a benefits of Data Control Plane. Set your data free on its journey while still maintaining viability, traceability and control.

Thursday, August 15, 2019

Data Lake vs Data Warehouse


Is a data lake part of your data warehouse platform or does the data lake sit beside it? There is a fair amount of ambiguity as to what a data lake is and how it should fit into your overall data strategy. 

I believe data lakes (coupled with elastic cloud storage and compute) are a game changer in both the DW and BI world. Your data warehousing strategy should be part of the data lake not the other way around. While you don't have to throw away everything you have done or learned in your traditional ETL and DW world, the fundamentals have changed. 

To take advantage of your data and build better BI/analytics you must build atop a sold data lake foundation. And this going well beyond the many failed Big Data and Hadoop projects of the recent past that many enterprises have experienced. 

While Hadoop was a necessary step forward at the time, it was and is an evolutionary dead end - RIP Hadoop. Cloud data lakes are the future and it is more than putting your data into S3 buckets. 

Well architected data lakes are the culmination of a succinct data management strategy that leverages the strengths of cloud services and many traditional DW best practices and data governance policies.

Wednesday, May 15, 2019

R.I.P. HDFS | The Cloud Wins!


HDFS is an evolutionary dead end in the tree of big data. Data lakes based on S3 object storage deliver on the promise of separating storage from compute and make it possible to scale your processing and downstream analytics/AI and data marts on top of a data lake in an agile and elastic fashion. The HDFS architecture always bugged me when it was first released (besides the fact it is written in Java). Moving the code to the Hadoop data node (usually only three replicas available by the way), seemed to be inherently limiting to me. It was not really better than using big unix SMP servers other than you got to use cheaper commodity hardware and grow incrementally. Good stuff, but not good enough - 1 step forward and a half step backwards.
While the idea of moving code to the data sounded cool at the time, it is fundamentally a bad data processing design for a truly scalable data lake that allows for rolling up an arbitrary number ephemeral compute clusters on top of your storage. There is a place for HDFS and traditional Hadoop clusters, if you have big fixed and slow evolving predictable cluster of compute/storage environment. For the rest of us, a cloud based data lake architecture will win in the end and allow for agile development to meet the fast paced needs of downsteam today's BI, analytics and AI/ML applications that need to sit on top of the mythical data lake.

Thursday, August 9, 2018

Choosing Between Spark ML, scikit-learn, and DNNs


Now these aren't the only considerations when deciding on how to build your data science stack and the related tooling you will need around it,  but it is a place a lot of organizations tend to begin their opening questions. Sometimes the answer may be, all the above. But you have to first reflect on your organizations goals and the level of your investment in any transformation effort, especially one that involves such a fundamental shift in how you to turn data into business value.

There are a number considerations that can influence your data science architecture that should be examined before establishing your AI platform. They include:

  1. ETL and data prep tools? AI does not work without data. Find it, mine for it and create it.
  2. Cloud, on-prem or hybrid for building your data science stack?
  3. How big is your data? Really how big is your data? Not everyone has "big data".
  4. What are you modeling? What kind of outcomes are you looking to solve for?
  5. Build, buy, partner. What kind of skills do you want to invest in for in-house data science, ML engineering and ML operations?
The bullets above are only touching on much deeper considerations that need to be assessed by any organization looking to transform their business with AI. But let's step back a bit and just discuss the question posed by the title of this blog to avoid turning this blog into a long drawn out analysis that goes down too many rabbit holes.

Spark ML
It is natural for a lot of organizations who have been doing "Big Data" to get their first exposure to data science through Spark's MLlib. Spark ML is a nice module/framework the comes with Spark and comes packaged with most major Hadoop distributions. The ML APIs and algorithms include many of the popular model building options from decision trees, to survival analysis (time-to-live), to allowing you to build recommendations engines (ALS), to unsupervised learning with clustering and topic modeling. Spark ML is nice and convenient for those coming from the Big Data universe. One nice advantage is that you can often leverage Spark's inherent distributed architecture to build models that can operate at large petabyte scale when needed. Is Spark ML ideal for all data sources and outcome objectives and it is the most efficient (you can hack DNN into if you have the stomach for it) - the answer as you might guess is obviously no. Why, well that is for another day to dive into, but suffice it to say that it may always be the most accurate way to build models and may not always be the best bang for CPU/GPU buck.

scikit-learn
Then there is good old scikit-learn. Any Python developer with a math or data background or has done any statistical modeling (or ML work) will know and likely love scikit-learn and all the other related Python packages such as numpy, scipy and pandas to name the most popular. scikit-learn is a treasure trove of algos and APIs. It is an awesome framework for ML developers and data scientists. Does it scale in same ways that Spark can - unfortunately no. But do you always really need it to? Look at your data before you answer that.

Deep Neural Nets
Then there is the new kids on the block, DNNs (back from the future). Tensorflow and PyTorch just to name a couple of the most popular are claiming to be universal function approximators that can model anything and solve for everything. Note, you will need to bring data and lots of it. They are data hungry. They can solve anything from classifications to generating word embeddings to creating generative models. There isn't much a DNN and its offshoots can't do theoretically. Through their natural fit with GPUs, they can scale fairly efficiently, and you can sometimes sort of distribute them with some extra heavy lift.

Taking your Models Live
A lot of what we just reviewed is about building and training models. Now, how do then take what we just trained and turn it into a service that predicts, classifies or generates data? That is also a topic unto its own. Operationalizing machine learning models can be non-trivial but it can also be not so difficult at times. It just depends on the model you are creating. For example, sometimes discrete bounded models can just be exported into a database, but often times the solution (input and output space) is not finite and requires creating distributed your build models as inference engines - and that is a bit more work. Then there is the nagging issue of how and when to update your models. Again another subject all together.

Buy or Build
So should you build or buy? The big boys (google, aws, azure) are all making a lot of what we just described available as MLaaS offerings (to various degrees of completeness). So stay tuned and current as the AI  technology world is changing fast.


Saturday, February 10, 2018

Don't Build your NLP Bot like a GUI


There has been an ongoing buzz about conversational user interfaces (CUI) and how they can enhance human-to-computer interaction. Many of us have experienced them to some extent both in play with voice interfaces such as Siri and Alexa and at work with messaging interfaces in tools like Slack. When CUIs are infused with a good dose of NLP (and context aware, machine learning..etc), they have the potential to become more than just a primitive command line or simple minded messaging/voice interface. The potential for building NLP powered virtual assistants and benefiting from the productivity that they can provide does exist, but you will have to get out of your comfort zone as a developer.

The Mental Leap to CUIs

Building CUIs using NLP and AI can take some getting used to when you have spent your career or education building graphical user interfaces (GUI). Be wary of getting caught in the trap of building your natural language interactions like you would build a GUI. A virtual assistant, like how some NLP bots function, can behave in many of the same ways you would interact with another human that would be working on your behalf. Now, a GUI is a very capable and functional interface, but is not a virtual human. And you have to keep reminding yourself that the bot you are building needs to behave like a virtual human, because it is easy to get caught in the trap of trying to build your NLP virtual assistant like you would a traditional GUI. It ends up being the worst of both worlds if you end up doing that.

Not that there is Anything Wrong with GUIs

Now there is nothing wrong with GUIs. For some tasks and functions they will always be superior to natural language. But for many things we do in our daily lives, language is a superior medium especially if the engaging entity is intelligent.

Fundamentally how you would ask (via text or voice) a virtual assistant to perform an action or make an inquiry on your behalf is different than using a GUI. The way you reach a decision or action point in a CUI can be different than a GUI. Let's take a very simple example to compare.

Say you are using a business application to manage pending requests that you must take action upon on a daily basis. And there are different types or classes of requests you must review. Some requests are specific to you, some to your direct team members, and other requests are company wide actions you must review, but you are required to review and make decision/action on all of them at some point during the day or week.

In a GUI you might navigate to the screen and see some quick filters that let's you select which requests types to view or all requests might be shown in some sorted of grouped order on a scrolling page. And you would navigate through the information to decide what to do first. There are many ways to represent the request and perform filtering and sorting. It also depends on the your preferences for what tasks you like to tackle first and how you manage your day. The ways to make the GUI optimal for your particular usage pattern depends on many factors, many of whom are specific to you. This is where GUIs begin to breakdown. They can sometimes overwhelm us with information or not adapt to our unique usage patterns - basically they lack the ability to easily adapt to the extreme personalization you can achieve with a CUI.

All Things being Equal CUIs Win

Now this all depends on your implementation of the GUI and CUI. You can obviously build a horrible CUI and a wonderful ultra personalized GUI, but all things being equal I claim that a CUI will always be superior. The advantage the CUI has is that the user experience does not have to change significantly to improve personalization. The inherent nature of conversational interaction is something natural to all humans. So as a developer, as you improve the conversational flow of your virtual assistant, and as you release new versions of the bot, the user can adapt much more easily because the flow is fundamentally the same, the interaction is still a conversation and it is the bot that is getting smarter/better and always assisting you through the same conversational experience to help you accomplish your objectives.

So let's go back to our request/action application we described earlier. In the CUI case, a smart virtual assistant might look at all the pending requests and tell you have many pending requests of different types and suggest you start with your direct reports first, since there is only one of those, if that was the case. The ability of the CUI to branch off in different directions is much more dynamic than what a GUI can do, and this can be done in a way that does not force the user to learn a brand new interface with each new release of the software, since the virtual assistant is your interface and your personal guide at the same time. The CUI sort of has built in help since it is a virtual assistant by design.

Mixed Mode CUI and GUI

Many of the popular bot platforms such as Slack and Facebook Messenger have incorporated some very convenient visual GUI components into their messaging and flows which allow developers to mix conversation question/answer dialog with short-cut GUI actions and interactions. This can be great way to meld conversational with GUI, but at the same time it can get CUI developers sucked back into the GUI world, and get developers focusing too much on injecting too many GUI interactions into the CUI flow. So use these tools wisely and keep the interactions focused on the conversational flow between the user and the human like virtual assistant.

Grow your Bot Like You are Raising a Child

So be careful, always think like you are building a smart virtual assistant not a GUI. You will not have all the smarts built into the bot from day one, but make that your mission to keep the conversational flow focused on the interactions and dialog between the human user and the virtual assistant, and everything else will follow as your bot gets smarter.

Saturday, February 3, 2018

Can AI Put the Human Back into HR?


Over the past couple of decades Human Resources (HR) has moved from paper and manual processes to more automation and providing employees with more and more web and mobile powered experiences. But has HR lost its human essence and personalization in the process? Nowadays managing a company's "human" assets seems to involve less and less human interaction between employees and the actual people in the HR organization. Try finding an HR person when you need one these day!

In the Beginning there were Humans

I recall my first few jobs at both major corporations and at startups, there was always a HR person I could reach on the phone, or simply walk over to their office to get small and big matters resolved. HR personnel where typically visible and accessible. Human Resources representatives interacted at a personal level with employees and often knew you on a first name basis.

It seems with more technology that HR has lost its human to human interaction and become more impersonal and mechanical. Today, if you need some payroll or benefits issue resolved, you typically need to submit a "ticket" in some online system and wait for someone (you have never met) to contact you back over email or if you are luck over the phone. Or if you are lucky you can click your way through a labyrinth of HR GUI applications to find what you need.

Virtual Assistants are the New Humans

So what is the solution, more or less technology? Maybe the answer is better technology. AI powered HR virtual assistants have the potential to be your personalized guide to do everything from answering general HR questions to requesting time-off and helping guide you to finding the information or actions you need to take quickly. Machine learning powered intelligent assistants could help resolve problems and questions with your payroll, for example. Interactive NLP powered bots could help guide you through, an often times, complicated benefits and open enrollment process by knowing your preferences and history to match you with the optimal recommendations for your particular situation.

The Future is an AI Powered Voice and Messaging-First World

The future of HR is not more technology, but more intelligent technology powered by machine learning, natural language processing, personalized recommendations engines, and other AI enabled technology that bring hyper-personalization and a human-to-computer interaction model that goes beyond impersonal graphical user interfaces. Voice and messaging-first interfaces (endowed with NLP) are a step in the right direction and can bring back a bit of humanity to your technology overloaded workplace.

Monday, December 25, 2017

Recommending Actions for Your NLP Bot

At first glance, the machine learning methods typically used in NLP applications (such as chatbots) and those used in recommender systems (for recommending products) are not often leveraged together in the same applications.

NLP is the machine learning domain that makes your virtual assistant capable of engaging in human language based conversation and recommender systems, as the name suggests, recommend products/services you will hopefully like (thus saving you the trouble of discovering them on your own); but it is not often you see NLP and recommender systems together.

Where Conversational UI Meets Recommendations

But let's think about that for a moment. Is there a solution space where NLP and recommender systems intersect and why would anyone want to do such a thing? I will make the case that every so called AI powered virtual assistant (aka chatbot and their kin) needs context and part of that context can be provided by a personalized recommender system that helps guide the conversation and streamline the conversational user experience.

A Messaging First World

We are in the midst of a messaging application revolution. A new generation of users are making messaging based applications their preferred medium for communicating with people, places and things around them, especially when it comes to the digital world (and contextual world). And there is no lack of applications from fintech to social applications leveraging and rediscovering the command line interface as the new mode (or not so new mode for many command-line geeks) of communication between humans and computers.

Your Next Question or Answer is a Recommendation Away

Ok, so where do recommender systems fit into the world of NLP and conversational driven user interfaces? Well, conversational applications are not without their own challenges. Typing and speaking takes effort on both the user and virtual assistant, in order to engage in timely and efficient interaction. But what if your virtual assistant new what you wanted to do next (or what you might/should like to do next)? What if your NLP powered bot could suggest to you actions you might want to take and save you the trouble of verbalizing it - maybe give you a quick one click shortcut (can be voice powered as well) to driving and continue the conversation?

This is where recommender systems can play a vital role in making your virtual assistant not just clever at understanding intent and named entity recognition from voice or text, but also present a sense of intelligence in remembering your past behavior or predicting what you might do next (or should do next) by relating your behavior to what others in similar roles and situations have done next. So even with no prior knowledge of "you", the virtual assistant might prescribe next actions based on what others in similar roles and situations have done. Does that not sound like a recommendation system?

Prescribing vs Predicting

Recommender systems are inherently about prescribing things (which can include actions not just items) applicable to your context at a given point in time (time based being a critical context as well). I foresee a future where both business and consumer application oriented virtual assistants and NLP bots will leverage highly personalized recommender systems to take the human-to-computer interaction to the next logical evolution (as promised by many sci-fi books and movies :)

Matrix Factorization and LSTMs are Your Friend

So for all you NLP bot developers, make things like matrix factorization and collaborative filtering your friend. Hybrid recommender systems based on collaborative filtering and content filtering (product and customer meta data) have been the state of the art for the past few years (since the Netfix contest). However the future of recommender systems will be powered by deep learning and concepts like LSTM and product and item embeddings. Research in this space is evolving fast. A mix of shallow and deep learning techniques are racing to enable this world of intelligent NLP bots and efficient conversational user interfaces.

Wednesday, September 13, 2017

AI and Machine Learning Madness


The hype around machine learning and AI is just off the charts these days. This reminds me of several years back when just about every software product started putting the term "Big Data" in front (or at the end) of all of their product literature and company tagline.

AI is the Art of Using ML at the Right Time and in the Right Context

Today, startups are claiming to change every industry and the world just because they put some "machine learning" inside their products. Machine learning has the potential to give software super powers, but alone it is does not create great software.

Listen, I am just as excited as the next geek about the potential of deep learning and the democratization of data science, but please stop putting machine learning in your company's slogan, like it was the secret ingredient that you were looking for. Sorry, this is becoming my new pet peeve.

Machine Learning, at the end of the day, is just one more tool in our arsenal of tools and skills needed to build more efficient and more intuitive solutions and services for our customers and for humanity as a whole. Infusing machine learning into your products is more than technology. It takes a new way of thinking and new of running engineering teams and engineering processes. 

A Great Product is Much More than AI

Please talk about the benefits of using AI/ML/DL, don't just say "because you are using machine learning" you are changing the world or have a better product. There is that little thing called building a product that delivers value to the end user that you still have to get right. And believe it or not, this still requires human creativity and execution. Claiming your infusing your product with ML will not necessarily make it a better product or the world a better place.

AI is in the Product - It Should not be the Product

I do believe that machine learning related technology will be in every future product, just like we have transistors in every computer today. Machine learning will allow us to build better and more efficient products. How you use the machine learning to build a great product will matter much more than the fact that you are using machine learning. Keep that in mind when you come up with your next startup tagline and company slogan!

End of rant.

Monday, August 28, 2017

AI vs Paradox of Choice


The paradox of choice is a problem we see more and more of in our modern world. It goes beyond what products Amazon should recommend or friends Facebook should suggest. In the business world and in enterprise applications this is also a challenging problem as our applications and processes grow in complexity. The potential for machine learning powered recommender systems to augment human decision making is one of the next frontier for AI in the enterprise . 

Recommender systems can do more than just suggest what articles you should read on Linkedin or what jobs are most suited for you. In the future machine learning (and more likely deep learning) powered recommender systems will guide enterprise decision making by helping business process owners take the most effective actions and decisions in a timely manner and with hyper-personalization. 

Recommender systems will move from solving B2C optimization problems (how they are typically used today in our data saturated and over marketed world) to solving problems in B2B and enterprise applications. Ultimately recommender systems are about prescribing (they are not really about predicting) an optimal decision at the right time and place/context, so they can naturally deal with a variety of B2B scenarios such as optimizing workflow paths, streamlining supply chain actions, to augmenting human decisions for common day to day business operational functions. Enterprise decision makers are in vital need of these AI super powers. Stay tuned they are coming :)

Sunday, August 20, 2017

Messaging-First Applications with Slack or FBM?


Conversational UX design is evolving as more and more apps begin to incorporate conversational UI functionality. While the concept of a messaging-centric UI can seem simple, the melding of a messaging-first user experience is nothing to underestimate. Conversational user interfaces can be simple for humans to interact with (you are just chatting back and forth), however, blending in and balancing rich visualization and complex interactions is not simple to get right. Just like any other UX, it is a balance of minimalism while allowing for rich expressiveness in the UI without overwhelming the user.

Slack is one of the leading platforms for building bots, especially for enterprise applications. However Slack has a number of bot conversational UX features that are still missing relative to other platforms such as FB Messenger and FB Workplace.

To give some perspective, here is my compiled list of features I would like to see in Slack's bot framework to improve its messaging UX and bring it on par with platforms like FB Messenger:

1) Conversational Streams and UI Alignment

Slack bots (especially in direct-messaging one-on-one dialog flows) force the bot and the user to both be left justified in the messaging UI stream. This goes against UI norms found in the majority of messaging application and related best practices for messaging apps. Typically in a streaming messaging flow, your conversational stream (you being the person interacting with the bot) is on the right of the screen and the party you are talking to (in this case the bot) is on the left side of the screen (or it can be visa-versa).

This is something not supported in Slack and makes a number of things awkward and cluttered in a bot-to-human dialog, especially when it is one-on-one (as opposed to Slack group channel). In Slack the entire conversational interaction is left justified, which can make the UI look cluttered when there are visual rich elements involved and and things like "Quick Replies" in the back and forth stream.

I hope that Slack will allow for aligning the bot vs the user on different sides of the messaging stream, something more similar to how FB Messenger works. This will allow for a more natural conversational interaction.

2) Horizontal Scrolling Carousel UI Components

Slack (mobile and desktop/web) does not provide any kind of horizontal card or horizontal scrolling carousel. While some might consider this bad design (to allow for horizontal scrolling of cards), it is often necessary to minimize the vertical area needed to display information in rich messaging interactions. FB Messenger allows for limited horizontal scrolling carousel that I find to be very useful when building bots. Hopefully Slack will incorporate this. Slack already supports rich "attachments", so it would be a natural fit to allow for some limited level or horizontal scrolling.

3) WebView Integration

Slack does not have explicit support for messaging buttons that open a webview UI. Sometimes a webview is needed to show rich web content (again here this kind of feature should not be abused). FB Messenger has this ability and allows for controlling how the webview window is opened and closed. This can be mimicked in Slack by using embedding links in the "field" elements for example, but is a bit of a hack.

4) Quick Reply Buttons

One particularly nice feature I got accustomed to using in Facebook Messenger is the feature referred to as "Quick Reply". This allows the bot to display "Quick Reply" buttons that are shortcuts for the user to enter commands that they would normally have to type.

There is a away to mimic quick replies in Slack, but again it is a bit of hack. Check this open source node/slack project for an example of how this works with Slack. Quick replies are a real necessity in a rich messaging interaction. Again here, I hope that Slack adds this feature natively instead of making bot frameworks jump through hoops to emulate this feature.

Hopefully the Slack product team will address these issues as Slack is by far the best team and enterprise collaboration/messaging platform on the market today.

FB Messenger might have some superior bot-to-human interaction and UX capabilities, but it inherently lacks the team collaboration functionality and the many third-party integrations that Slack has to offer.

I do believe FB Workplace will close the gap over time, and in many ways has advantages over Slack in terms of out of the box social collaboration functionality. Slack is a bit of geeky technical tool when it comes to social collaboration and thus not as intuitive to use.

I expect both FB Workplace and Slack to evolve as head to head competitors and battle for the hearts and minds of developers much like how Netscape battled Microsoft's Internet Explorer for web domination. For enterprise owners and enterprise end user, intelligent AI endowed virtual assistants and bots will usher in a new era of innovation not seen since the dot-com days. The battle has moved from the mobile app store to the AI app store where natural language understanding and deep learning are the killer technologies in the arsenal of AI sophisticated developers.

Thursday, July 13, 2017

Augmented Reality vs Conversational User Interface



Don't believe all the hype about virtual reality and its close cousin augmented reality. How many examples do we need to see before we get it that humans don't like a big gizmo sitting over their faces and eyes or attached to their body. Examples like the failed Google Glass and the failing Facebook Oculus are two examples of such failures, with more on the way if Apple is not careful.

What will win out? It's all about texting and voice stupid (not you). AI is surely coming, but it will be powered by voice-first and messaging-first applications and services. Visual augmentation with AR and VR, is cute, but it will not be what transforms our reality and changes how we interact with technology.

Humans are social animals and voice and texting is what taps into that part of our brain that drives us to connect with others/things and drives us to share ideas. So gear up and start thinking how to turn your applications and services into voice-first and messaging-first user experiences. That is where the future is going.

Monday, June 26, 2017

Conversational Bot UI Features Missing in Slack


Conversational UX design is evolving as more and more apps begin to incorporate conversational UI functionality. While the concept of a messenger-centric UI can seem simple, the melding of a messaging-first user experience is nothing to underestimate. Conversational user interfaces can be simple for humans to interact with (you are just chatting back and forth), however, blending in and balancing rich visualization and complex interactions is not simple to get right. Just like any other UX, it is a balance of minimalism while allowing for rich expressiveness in the UI without overwhelming the user.

Slack is one of the leading platforms for building bots, especially for enterprise applications. However Slack has a number of bot conversational UX features that are still missing relative to other platforms such as FB Messenger, for example. Here is my compiled list of features I would like to see in Slack's bot framework to improve its messaging UX and bring it on par with platforms like FB Messenger:

1) Conversational Streams and UI Alignment
Slack bots (especially in direct-messaging one-on-one dialog flows) force the bot and the user to both be left justified in the messaging UI stream. This goes against UI norms found in the majority of messaging application and related best practices for messaging apps. Typically in a streaming messaging flow, your conversational stream (you being the person interacting with the bot) is on the right of the screen and the party you are talking to (in this case the bot) is on the left side of the screen (or it can be visa-versa).

This is something not supported in Slack and makes a number of things awkward and cluttered in a bot-to-human dialog, especially when it is one-on-one (as opposed to Slack group channel). In Slack the entire conversational interaction is left justified, which can make the UI look cluttered when there are visual rich elements involved and and things like "Quick Replies" in the back and forth stream.

I hope that Slack will allow for aligning the bot vs the user on different sides of the messaging stream, something more similar to how FB Messenger works. This will allow for a more natural conversational interaction.

2) Horizontal Scrolling Carrousel UI Components
Slack (mobile and desktop/web) does not provide any kind of horizontal card or horizontal scrolling carousel. While some might consider this bad design (to allow for horizontal scrolling of cards), it is often necessary to minimize the vertical area needed to display information in rich messaging interactions. FB Messenger allows for limited horizontal scrolling carousel that I find to be very useful when building bots. Hopefully Slack will incorporate this. Slack already supports rich "attachments", so it would be a natural fit to allow for some limited level or horizontal scrolling.

3) WebView Integration
Slack does not have explicit support for messaging buttons that open a webview UI. Sometimes a webview is needed to show rich web content (again here this kind of feature should not be abused). FB Messenger has this ability and allows for controlling how the webview window is opened and closed. This can be mimicked in Slack by using embedding links in the "field" elements for example, but is a bit of a hack.

4) Quick Reply Buttons
One particularly nice feature I got accustomed to using in Facebook Messenger is the feature referred to as "Quick Reply". This allows the bot to display "Quick Reply" buttons that are shortcuts for the user to enter commands that they would normally have to type.

There is a away to mimic quick replies in Slack, but it again, it is a bit of hack. Check this open source node/slack project for an example of how this works with Slack. Quick replies are a real necessity in a rich messaging interaction. Again here, I hope that Slack adds this feature natively instead of making bot frameworks jump through hoops to  emulate this feature.

Hopefully the Slack product team will address these issues as Slack is by far the best team and enterprise collaboration/messaging platform on the market today.

FB Messenger might have some superior bot-to-human interaction and UX functionality, but it inherently lacks the team collaboration functionality and the many third-party integrations that Slack has to offer. Perhaps FB Workspace might have something to offer down the road, but it has a long way to go in order to catch up with Slack.

Tuesday, June 20, 2017

Building Slack Bots with Node.js

There are a number frameworks for building Slack bots such as botkit and beepboop. But sometimes you just want to use the native Slack APIs and build your bot without the added burden of a complex bot application framework. If this sounds like what you are looking for, then you have come to the right place.
This project demonstrates how to apply the essential Slack APIs using the keep it simple principle and a straight forward approach to extending and customizing your bot's conversational dialog and interactive UX (interactive buttons/menus).
Slack has an extensive set of APIs for building user-to-bot conversational UI and interactive messaging applications. This project implements a bare minimal bot application written in node that provides a template for integrating with the various Slack APIs. This is a starting point application for building your bot that can be thought of as a basic modular template and set of coding patterns for building a Slack bot that exercises the essential Slack APIs and bot interaction features.
Support for api.ai is included for demonstration purposes, but other NLP services can be used instead.
The focus of the project is to make it easier to implement NLP intents and conversational bot/user interactions and the associated UI interactions that might come about from the use of Slack attachments (such as Slack interactive message buttons and interactive message menus). Adding new NLP intents and handling the callbacks for interactive UI elements are demonstrated in this basic bot. Examples for using incoming webhooks and slash commands are also provided.
The hope is this project will make it easier to start building a Slack bot by providing a template on which you can extend your bot with new dialog/conversations and interactions. The coding patterns will hopefully make it easy to extend the conversational user interaction when building your bot.
By design, this bot uses the Slack Event API and avoids any use of the Slack RTM API.
Note, that this bot was not designed to work with Slack Enterprise Grid. Some minor modifications would be required to have it work in a cross team enterprise environment.
Slack APIs demonstrated in this bot application:
  • Event API (subscribes to: "message.im" and "reaction_added").
  • Sending Text/Attachment/Interactive Messages via channel reply.
  • Sending Text/Attachment/Interactive Messages via Incoming Webhook.
  • Handling Slash commands.
  • Handling Interactive Message actions.
  • Handling oauth.access to store and use bot access token.
In addition to the core Slack REST APIs, this app demonstrates how to use the following slack wrapper APIs:
  • @slack/client
  • @slack/events-api
  • @slack/interactive-messages
  • @aoberoi/passport-slack
Give the GitHub project a spin. Hopefully you will find it a good starting point for building Node native Slack bots.