The Future of Online Shopping and Digital Fashion
/Women Leading Visual Tech: Interview with Zalando’s Senior Data Scientist Nour Karessli
LDV Capital invests in people building businesses powered by visual technologies. We thrive on collaborating with deep tech teams leveraging computer vision, machine learning, and artificial intelligence to analyze visual data. We are the only venture capital firm with this thesis.
Our Women Leading Visual Tech series is here to showcase the leading women whose work in visual tech is reshaping business and society.
Nour Karessli is a Senior Data Scientist at Zalando, Germany’s multinational e-commerce company. She is utilizing state-of-the-art computer vision and AI techniques to leverage the visual cues present in fashion images to better understand the complex problem of size and fit.
Additionally, she sits on the organizing committee of the Zalando Data Science Community and is responsible for organizing weekly community sessions and an annual conference called the Zalando Data Science Days. Nour is also co-organizing the Women in Computer Vision workshop held in conjunction with CVPR 2021.
Nour holds a Bachelor’s degree in Computer Engineering focused on Software Engineering from Damascus University. She completed her Master’s in Computer Science from Saarland University focused on machine learning and computer vision.
Karessli started her career as a Research Assistant at Deutsches Forschungszentrum für Künstliche Intelligenz and Max Planck Institute for Informatics. She worked on the “Effekt project” that aims to aid users in choosing the optimal bicycle route using the shared information among cyclists. Later, she joined EyeEm, a global photography community and marketplace. As a Computer Vision Engineer, she was working on low-shot learning for image classification using concept embedding and machine learning.
Abigail Hunter-Syed discussed with Nour the future of online shopping and digital fashion. (Note: After five years with LDV Capital, Abby decided to leave LDV to take a corporate role with fewer responsibilities that will allow her to have more time to focus on her young kids during these crazy times.)
The following is the shortened text version of the interview.
Abby: How do you describe what you do in one line?
Nour: I'm an applied scientist at Zalando. We're working on machine learning and computer vision solutions to help customers get the right size and fit from the first time that they buy items.
Abby: How much of a problem is size and fit for online shopping?
Nour: When you go to stores and you try things on, this is the process of checking things up, and this is missing in online shopping. Even in stores, you probably get 2-3 sizes of the same item to find the right one. Online, you only have the photos and a few descriptions of the items. You can try to relate to how the model's body looks in comparison to yours, but usually, you have this delayed feedback until you get the item, feel the fabric, and see how it's going to look like on you.
Around 50% of the items that are purchased online get returned. About 1/3 of these returns are caused by “wrong” size and fit.
Abby: At least as a starting point from an e-commerce perspective, having people be able to virtually try on these things, is that where you're going? Or do you think that's going to stay more in matching people between the items and their current size and what they think the supposed size is?
Nour: It's a mixture of many solutions. There's a term that we use in the team, we call it ‘a mountain peak’. There are many ways to get there. What we are trying to solve is not going to be served into one algorithm at the end. There will be multiple solutions that can be served based on the context – who the customer is, how new they are to the platform, what we know about them and what they are willing to share with us. It also depends on the item category. There are many pieces to the puzzle to pick the right solution.
Abby: Have you always known that you wanted to pursue a career in deep tech or was it the fashion side of things that was more intriguing to you at the beginning?
Nour: It's tech. When I was a kid, I was into video games. I used to spend hours on video games with my siblings solving puzzles.
Abby: Which one was your favorite?
Nour: The Sims was my favorite. I liked building houses, making people fight and love each other.
In high school, I got into coding. It was almost the same time when the social media platforms hit off. I got interested in web development and how things were built and connected. Then I decided to study computer science. I was focused on software engineering but then I shifted in my master’s to AI and computer vision.
Abby: Did you have a professor or somebody that you looked up to as a role model who helped you make the switch or in general fostered your interest in the AI side of things?
Nour: I was inspired by the people that I'd seen. We had many brilliant professors when I was doing my master’s. The mixture of courses that I had, slowly led me into this path. In the beginning, I was interested in user experience, how people perceive things and also some visual understanding and the cognition of the interfaces. I was also interested in AI, how it works, how we can control robots and agents. It all combined when I started to work on my master’s thesis. It was on the topic of understanding the human gaze – information that helps machine learning zero-shot classification tasks.
Abby: Can you give me an example of what that would be? What was one of the things that you looked at to understand the human gaze better?
Nour: When you're talking to a kid who has never seen a zebra, you can describe it by saying it's similar to a horse, but with white and black stripes. Even without seeing any, they would imagine how it would look like. The idea was to use a similar method, but instead of having textual description is using the gaze information.
We had an experiment where we recorded participants' gaze – how they look at the different classes. We had the set up like a game that you usually see in the newspaper, where you get two similar items and you try to find the differences. We had something similar and asked people to find the correct class, the category. From their gaze pattern, we tried to build category descriptors and use that for machine learning.
Abby: What was an example of something that you were looking for people to be able to correctly identify?
Nour: It was mostly birds, cats and dogs. It was also specific for fine-grain classes, we were trying to differentiate woodpeckers or sparrows for example.
Abby: What did that teach you about human intelligence that you need to replicate with the algorithms that you're building?
Nour: Our vision system is fabulous! It’s amazing how we perceive and recognize things in a short amount of time. It was interesting to see how people note different features of the category and make decisions. Our experiments showed that it's not one method. You have different methods and if you combine them from different people, you get better descriptors because each one of us has our kind of algorithm in our mind.
Abby: How did you transition from looking at gaze analysis into the future of fashion? What was the most intriguing for you about it?
Nour: Fashion is a big part of our life, even if we don't realize it or spend too much time on it. When I joined Zalando, fashion was one of the buying things for me because I thought it's personal for people's lives and I wanted to have this kind of impact on people. You can do a lot with computer vision but some solutions and applications are very impactful, and I thought I want this huge, intangible impact on people's life. That's why I thought a corporation like Zalando, which has millions of customers, is the right place to go.
Abby: That's such a neat perspective that it is a personal thing. The moment that we go into the dressing room, if we're in a store to try things on, is very personal. You don't want anybody watching you as you're in there and half the time you don't even come out with the things down to show whoever it is that you're shopping with. Figuring out the personal relationship with e-commerce, getting into what you're talking about a little bit earlier, in terms of people not necessarily being willing to share the data or their information. How do you overcome that, or how do you say, ‘hey, it's worth it for you to share this information because I can tailor and personalize this experience for you’?
Nour: There are multiple ways to tackle this. First, start to share information to build trust. When we give size advice, we don't do it only on a personalized level. We have different types of recommendations and one of them is article-centric, so we don't tell the customers what they should buy or what exact size they need to pick, we tell them what we know about the items.
We have a lot of items. We have a lot of data. On a large scale, when we share this, it starts to build trust between us and the customers.
Then it comes to the dialogue. We have products to give the customers the space to express themselves. We ask them to describe their experience: ‘was the item too big or too small?’. They can also choose “it’s not for me”. This survey is simple and brief, our customers are not overwhelmed.
We have much more advanced solutions for more engaged customers. We ask them for more sensitive information about their usual sizes and measurements of their bodies.
To maintain the trust you need to be transparent about what information you use and how you want to use it to get this recommendation.
Abby: Of all of the customers that you have on Zalando, what percentage of them is actively sharing their information with you or giving you feedback?
Nour: Many people share at least why they're returning items. If they share with us the reason for their return, and if it’s due to size and fit, we start giving them more personalized recommendations.
Abby: When I used to live in Switzerland, I'd use Zalando a lot. I don’t use it anymore since I live in the USA but I still use ASOS. Don't be angry! ASOS automatically recommends me to size up or size down, it gives me a recommended size. That's still just based on my feedback in terms of the things that I've returned and the things that I haven't returned so there's not necessarily too much sophistication to it. Where does the actual vision component come in?
Nour: Three years ago, when I started at this company, the team was building a lot of algorithms around the return and purchase data so a lot of the models we have in production were based on these types of datasets.
We had this idea to use images of articles for size and fit, and at the time it sounded a bit risky to think that you can see the size and the fit of the items from the images. Clothes are usually put on mannequins or fashion models, the pictures are taken from different angles and they probably don’t show all the information that you're looking for.
Too big, too small, or buy one size up or one size down – this kind of advice requires some data for new articles. As soon as the new item is uploaded to the platform, you can judge it by more than its description.
Once you start getting purchases and returns, it might be a bit too late in the process to start to judge it. We had an idea to leverage these images and the visual data that we have on the articles to learn the relationship between the visual appearance of the garment and its sizing behavior. We developed a model that takes the article image and learns from the visual appearance of the item, from the interrelation between the dimensions of the item, if it's going to have some sizing issues or not. This model is in production. It's used as an early signal to provide size advice as soon as possible.
Abby: It sounds like your algorithm can pick up on something like the elasticity of an item. You know that it's going to be a tighter fit thing because it's got the elastic nature like these are the super-stretchy jeans versus structured ones. I can presume that also then takes on another challenge that e-commerce seems to have, which is that touch and feel component. One of the big drivers behind returns in e-commerce, besides the fit, is just the touch and feel. You ordered something and you thought that it would feel one way or it looked like it would be super-soft and fuzzy, but then you get it and it's wooly and tough. Do you see that as the next step for you guys to be able to do is to report on, what is the texture of these garments as well?
Nour: The texture is an interesting question but there’s one step before that. The whole fit of the garment is still unsolved at the moment. When you think about how things will look like on you.
Abby: Like the weight and the drape and all that kind of fun stuff?
Nour: Exactly. When you start thinking about a 3D person, the first thing that comes to your mind is to visualize these items on a 3D person, and see where it's going to be loose, where it's going to be tight, if it's going to be too tight, etc. You get some insights from what you know about the fabric. You can say, this is elastic so it's going to be elastic in some parts and tight in others. The first step is to get the 3D virtual fitting.
Abby: Instead of it being a filter that sits on top and has that uncanny-type look to it, did you want the garment to be able to fit somebody in 3D so you can see how it realistically would look, from a shape perspective?
Nour: The 3D person should be resembling the body of the customer.
Abby: It's always a disappointment when you've seen this dress on a beautiful model in the pictures, and you order and you're like, “why don't I look like that in it?”
Nour: Expectations vs. reality.
Abby: Do people want to see themselves digitally in a garment or would we much rather see a beautiful model in the garment and that helps us purchase it? Is our mentality changing a little bit around seeing ourselves virtually and seeing a replica of ourselves try things on?
Nour: It's not trivial for many people to think about how complicated it is to show the customers what their bodies would look like in front of them. We perceive ourselves in a different way from what the world perceives us. We only see ourselves with mirrors and from a different perspective. When you see yourself after being 3D scanned, it's usually like, ‘wow, do I look like that?’ It's not what you expect. It can be positive or negative but it's always shocking to see yourself from a different perspective. It's like when you record your voice. When I record my voice and hear it back again, it's always surprising.
We do a lot of user research and try to, for example, show them prototypes of how things will look and get their feedback. An important piece of our work is to understand whether people are willing to accept this kind of solution.
As you described, it's not always the best solution to show, and people can choose not to see that, so they would rather see it on a model and not on themselves. That's why I said at the beginning, there is no one solution because it's up to the person to pick what fits them.
Abby: Interestingly, over the past year everything has gone online and we sit in these Zoom calls all the time. There have been all sorts of metrics about a boom in facial reconstruction surgeries - chin tucks have gone up like an astronomical amount in the US - because people are constantly looking at themselves on the screen. It's one of the reasons why they think there's such mental burnout, but tying it to this, people aren't necessarily changing in the way in which they want to see themselves on screen. If I think about avatars as a whole in the fashion space, one of the things that I'm interested in is what is the role of digital fashion as we move forward? Considering that we exist so much as these digital versions of ourselves, how far off are we from being able to buy clothing that could fit us digitally, as we sit in on a call or something like this?
Nour: If you look at the filters and what people are using now with the backgrounds, I don't think it's far in the future that we start doing this, especially if we continue to be in this pandemic situation with the remote work and everything.
There's a big chance that we’ll start buying clothes online and just have them digitally.
A digital dress by The Fabricant
Abby: One of those examples, The Fabricant, is selling $9,500 digital dresses that are only made to be fitted to you in your Instagram feed. Is this going to be a bunch of graphic artists who are taking a digital rendering of a piece of clothing and fitting it to somebody, or is there the potential using some of these fit technologies, like you guys are developing, to automatically enable people to wear things for their TikTok dance or their Zoom call, or as they stream themselves on Twitch? Do you think that there's going to be a market for that, and is it technically feasible?
Nour: It depends on how realistic it's going to look. I would say technically it may not be realistic as of today but in the future, it's going to be a thing. The way I see it with fashion is that even if you exist digitally, it has to meet some physical requirements. At least you need to make sure that what you have from the fabric, from the cut, that it's feasible to manufacture this item and deliver it if customers want to. Even if we spend too much time in the digital world, especially with fashion, there might be moments that we want to materialize what we see and have it in real life.
Abby: As somebody who used to love gaming and The Sims, for instance, if you could shop on Zalando for a cool dress that you like and you choose it for your character in The Sims, and then buy it for your real-life person, is that something that you find intriguing?
Nour: Maybe not for me at this age but there is a market for it.
Abby: Are there any other applications that are computer vision or deep learning that are exciting for you at the moment?
Nour: When it comes to fashion, I find article generation interesting, especially what can be happening next. We can bring the customer in the loop so they are also part of the designing phase.
I’m talking about tailoring things to your taste, preferences of fit, or shape. Let’s say I might like someone's dress but in a slightly different fit or slightly different blend.
This can be the next thing for fashion and especially e-commerce because you can show things online and allow customers to play with them.
Abby: If you're able to customize specific elements of it within your shopping experience, that would be revolutionary. Okay, last question: if there were no computers that existed in this world, what do you think your career choice would have been?
Nour: I think I would be in the medical field. Having a doctor in my family, I was also tempted to go there but my love for computers was stronger.
In December 2021, we hosted a virtual event, “Visual Tech Radically Transforming Fashion” – check out our event recap and watch the video recordings on our YouTube channel.
