Method enables better control of GAN image generators' output

• Updated on: 2023-10-31
• Read original article here

Computer vision
Method enables better control of GAN image generators' output
By plotting nonlinear trajectories through a GAN’s latent space, the method enables certain image attributes to vary while others are held fixed.
Rayleigh EigenDirections (REDs): Nonlinear GAN latent space traversals for multidimensional features
Generative adversarial networks ( GANs ) are a technology that can produce remarkably realistic synthetic images. From a set of real images, a GAN learns a mapping from a latent distribution to the image distribution represented in the training dataset.
Modifying images by controlling GANs is a lively topic of research, whose applications include dataset creation and augmentation, image editing, and entertainment. Researchers have developed ever more sophisticated techniques for both exploring and structuring latent spaces, in order to understand how movement through the spaces translates to modification of synthetic images' properties.
In a paper that we presented at this year’s European Conference on Computer Vision ( ECCV ), my colleagues and I describe a new technique that offers precise control over GAN outputs. Unlike prior techniques, ours can hold selected image attributes steady — say, the location and appearance of one sofa in a room — while varying others.
In this sequence of images, we hold one feature of a GAN-generated image (the sofa, boxed in red in the first image) steady while varying the others around it.
Prior approaches to controlling GANs depended on linear trajectories through the latent space, along which some feature would vary — say, the age of the faces being generated, or the extent to which they were smiling or frowning. Researchers either looked for existing axes in a latent space, in which case the correlations with image features were rarely exact, or they intentionally structured the space so that it lent itself to linear trajectories, in which case they had to know in advance which image features they wanted to control.
Our method can find nonlinear trajectories through a latent space that hold some properties steady (in this case, the identity of a — nonexistent, synthetic — face) and vary other properties (hair length or color).
Instead of correlating spatial axes with predetermined features, our method plots a nonlinear trajectory through a GAN’s latent space. Consequently, it can work with existing GANs, regardless of the structure of their latent spaces. That means we can, in principle, control multiple arbitrary attributes.
By the same token, we can control features that would be difficult for humans to annotate accurately — and therefore difficult to capture by modifying the structure of the latent space. For instance, taking the Fourier transform of an image, we can fix the high-frequency characteristics and vary the low-frequency characteristics, producing clearly distinct images whose variations, however, are difficult to explain:
A source image (far left), followed by three images in which low-frequency characteristics are held steady, while high-frequency characteristics are varied, and three images in which the reverse is true. It would be difficult for a human annotator to label the differences between the images.
Finally, most work on controllable GANs has focused on synthetic faces, which simplifies the problem somewhat, since the same facial characteristics tend to inhabit approximately the same regions of the image. Our method, because it plots local trajectories through an arbitrary latent space, can handle more diverse types of images.
Rayleigh quotients
Our approach depends on the intuition that for any point within the latent space, there exist local trajectories in which desired attributes do not change. We treat the calculation of such a trajectory as an optimization problem — particularly, a Rayleigh quotient.
Related content
Explicit control of GAN-generated synthetic images
New method enables users to specify properties such as subject age, light direction, and pose in images produced by generative adversarial networks.
We assume that for any point in the latent space, there is a function that maps the corresponding image to some kind of feature set. In the case of features like hair length or eye color, the function would be a neural network, trained on the relevant classification tasks; in the case of high-frequency and low-frequency image characteristics, the function is a closed-form transformation like the Fourier transform.
The aim is to find a local trajectory through the latent space that minimizes variation in the outputs of some of those functions while maximizing variation in the outputs of others. Optimizing the ratio of those variations is an instance of Rayleigh quotient maximization.
We approximate relative displacements in the space using local linear expansions — linear approximations of a function’s value at a given point that are based on its derivatives. Assembling the matrix of derivatives — the Jacobian, which measures variation or rate of change along different dimensions — requires us to sample local points in the latent space. Once we’ve done that, the maximization of the Rayleigh quotient has a closed-form solution, which gives us the optimal trajectory through the space.
Related content
Growing generative adversarial networks, layer by layer
A new approach that grows networks dynamically promises improvements over GANs with fixed architectures or predetermined growing strategies.
We traverse that trajectory for a short distance, then re-compute a new Rayleigh quotient. The distance between waypoints is a hyper-parameter of the method, which varies according to function. In our experiments, we chose parameters that led to small but perceptible differences in the images corresponding to the waypoints.
In those experiments, we compared our approach to three prior approaches that found linear trajectories within the latent space, using GANs trained on two different datasets, one a set of faces and one a set of living room scenes. We found that, across the board, our approach did a better job than the baselines of both fixing the features to be fixed and varying the features to be varied.
Research areas
Raghudeep Gadde is a senior applied scientist at Amazon.
Related content
Igor Kviatkovsky , Alon Shoshan , Nadav Bhonker
October 15, 2021
New method enables users to specify properties such as subject age, light direction, and pose in images produced by generative adversarial networks.
Staff writer
October 26, 2022
Research topics range from visual anomaly detection to road network extraction, regression-constrained neural-architecture search to self-supervised learning for video representations.
Dylan Slack, Nathalie Rauschmayr
December 07, 2021
Applied Scientist, Ring
NL, Amsterdam
Job summaryAre you a passionate scientist in the computer vision area who is aspired to apply your skills to bring value to millions of customers? Here at Ring, we have a unique possibility to innovate and see how the results of our work improve the lives of millions of people and make neighborhoods safer.You will be part of a team committed to pushing the frontier of computer vision and machine learning technology to deliver the best experience for our neighbors. This is a great opportunity for you to innovate in this space by developing highly optimized algorithms that will work on scale. This position requires experience with developing efficient computer vision algorithms on resource-constrained computing platforms on edge. You will collaborate with different Amazon teams to make informed decisions on the best practices in machine learning to build highly-optimized integrated hardware and software platforms.Key job responsibilities* Research and implement the state-of-the-art computer vision and sensor fusion algorithms for resource-constrained computing platforms on a large scale.* Collaborate with product managers and engineering teams to design and implement computer vision and machine learning based features for Ring devices* Influence system design and product vision by making informed decisions on the selection of technology, data sources, algorithms, and sensors.
Senior Principal Research Scientist, AWS Networking - BERE
US, Virtual
Job summaryAmazon Web Services (AWS) is looking for a highly motivated and talented scientist to drive research into network performance between our cloud instances and to the general Internet. This work will lead the way to enable continuous and material improvements to our customers experience of the worlds largest cloud network.This person will not only identify where we in the network we need to go deep, they will prototype systems to demonstrate solutions and work directly with development teams to bring these solutions to production. They will act as part of the senior leadership team, within the global network connectivity team, while also influencing key services that utilize the network.
Applied Scientist, AdRisk
GB, MLN, Edinburgh
Job summaryAmazon is looking for an Applied Scientist to join the AdRisk team in Edinburgh. This team builds scalable solutions that monitor the technical security across Amazon’s advertisement portfolio. Our work is characterized by advanced Machine Learning techniques, high scale, complexity and a constant need for innovation.The ideal candidate will have proven experience employing cutting-edge machine learning to real-world problems to deliver results. Strong computer science and algorithmic skills, knowledge on web security, exposure to internet-scale businesses, working in cross-functional teams, and a track record of peer-reviewed publication in a relevant area are desirable.As an scientist working at Amazon, you will play a key role in identifying business opportunities, measuring opportunity, inventing and prototyping solutions. You will use a wide range of technologies, programming languages and systems, and work alongside an experienced engineering team. You will have the freedom and encouragement to explore your own ideas and the reward of seeing your contributions benefit millions of customers worldwide.
Principal Economist, PXT Central Science
US, GA, Atlanta
Job summaryThe Central Science Team within Amazon’s People Experience and Technology org (PXTCS) uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. We are an interdisciplinary team, which combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal.
Applied Scientist, Ring
NL, Amsterdam
Job summaryAre you a passionate scientist in the computer vision area who is aspired to apply your skills to bring value to millions of customers? Here at Ring, we have a unique possibility to innovate and see how the results of our work improve the lives of millions of people and make neighborhoods safer.You will be part of a team committed to pushing the frontier of computer vision and machine learning technology to deliver the best experience for our neighbors. This is a great opportunity for you to innovate in this space by developing highly optimized algorithms that will work on scale. This position requires experience with developing efficient computer vision algorithms on resource-constrained computing platforms on edge. You will collaborate with different Amazon teams to make informed decisions on the best practices in machine learning to build highly-optimized integrated hardware and software platforms.Key job responsibilities* Research and implement the state-of-the-art computer vision and sensor fusion algorithms for resource-constrained computing platforms on a large scale.* Collaborate with product managers and engineering teams to design and implement computer vision and machine learning based features for Ring devices* Influence system design and product vision by making informed decisions on the selection of technology, data sources, algorithms, and sensors.
Applied Science Manager, Sponsored Products Detail Page Allocation and Pricing
CA, ON, Toronto
Job summaryWE ARE OPEN TO HIRING THIS ROLE IN SEATTLE, TORONTO, NYC OR PALO ALTOAmazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering a collection of advertising products that drive discovery and sales. As a core product offering within our advertising portfolio, Sponsored Products (SP) helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The SP team's primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day!As an Applied Science Manager in Machine Learning, you will: Directly manage and lead a cross-functional team of Applied Scientists, Data Scientists, Economists, and Business Intelligence Engineers.Develop and manage a research agenda that balances short term deliverables with measurable business impact as well as long term investments.Lead marketplace design and development based on economic theory and data analysis.Provide technical and scientific guidance to team members.Rapidly design, prototype and test many possible hypotheses in a high-ambiguity environment, making use of both quantitative and business judgmentAdvance the team's engineering craftsmanship and drive continued scientific innovation as a thought leader and practitioner.Develop science and engineering roadmaps, run annual planning, and foster cross-team collaboration to execute complex projects.Perform hands-on data analysis, build machine-learning models, run regular A/B tests, and communicate the impact to senior management.Collaborate with business and software teams across Amazon Ads.Stay up to date with recent scientific publications relevant to the team.Hire and develop top talent, provide technical and career development guidance to scientists and engineers within and across the organization.Why you will love this opportunity: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate.Impact and Career Growth: You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.Team video ~ https://youtu.be/zD_6Lzw8raE
Applied Scientist, Amazon Marketing Cloud
US, TX, Austin
Job summaryAmazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering a collection of advertising products that drive discovery and sales. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day!Advertising Amazon Marketing Cloud (AMC) teams build Advanced Analytics System that are secure, privacy-safe, and dedicated cloud-based environment in which advertisers can easily perform analytics across multiple, pseudonymized data sets to generate aggregated reports. Inputs can include an advertiser’s own data sets, as well as their Amazon Advertising campaign events, such as impressions, clicks, and conversions. AMC reports can help with campaign measurement, audience refinement, supply optimization, and more, enabling advertisers to make more informed decisions about their cross-channel marketing investments. Our services ingest billions of behavioral signals every day. Speed, scale, and accuracy are critical to our success. As an Applied Scientist on this team, you will: Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity.Perform hands-on analysis and modeling of enormous data sets to develop insights that increase traffic monetization and merchandise sales, without compromising the shopper experience.Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models.Run A/B experiments, gather data, and perform statistical analysis.Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving.Research new and innovative machine learning approaches.Recruit Applied Scientists to the team and provide mentorship.Why you will love this opportunity: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate.Impact and Career Growth: You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.Team video https://youtu.be/zD_6Lzw8raE
Marketing Data Scientist
US, CA, San Francisco
Job summaryAbout Us:Twitch is the world's biggest live streaming service, with global communities built around gaming, entertainment, music, sports, cooking, and more. It's where millions of people come together to chat, interact, and make their own entertainment.We're about community, inside and out. You'll find coworkers who are eager to team up, collaborate, and smash (or elegantly solve) problems together. We're on a quest to empower live communities, so if this sounds good to you, see what we're up to on LinkedIn and Twitter, get interviewing tips on Instagram, and discover projects we're solving on our Blog.About the Role:Data is central to Twitch's decision-making process, and data scientists are an important part to promote data-driven decision-making in all of our operations. As a part of the Global Marketing Data Science team, you will help grow Twitch creators, attract and retain viewers, and measure our efforts to tell Twitch's story to the world. You will be a partner to your team, influencing the way campaigns are accomplished and their performance is measured, scaling analytics methods and tools, and leading the way for high quality, high velocity decisions.For this role, we're looking for an experienced marketing data scientist who will help develop marketing initiatives within our marketing team. You will define and track KPIs, design and measure experiments, create data dashboards, and inform decisions. You are a "full-stack" data powerhouse who uses data to guide decision-making. You will report to the Data Science Manager and be on a team that supports the measurement and direction of a variety of marketing activities, including email, social media, communications, product marketing, and performance marketing, among others.You can work in San Francisco, CA; Irvine, CA; Seattle, WA; New York, NY; and Salt Lake City, UT.You Will:• Establish analytical frameworks and provide expertise on measurement, data science, and experimental design to evaluate impact of marketing efforts on creators, viewers, and the entire Twitch ecosystem.• Surface key data to Marketing partners: perform ad-hoc analysis, build automated dashboards, and self-service reporting tools.• Write papers that explain, align, and evangelize measurement protocols across the company.
Economist I
US, WA, Bellevue
Job summaryThe Skunkworks team within PXT uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. We are an interdisciplinary team, which combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal.We are looking for an economist with expertise in applying causal inference methods, especially experimental design to topics in labor, personnel, education, or behavioral economics. We are particularly interested in candidates with experience applying these skills to strategic problems with significant business and/or social policy impact.The candidate will work with economists and engineers to estimate and validate their models on large scale data, and will help business partners turn the results of their analysis into policies, programs, and actions that have a major impact on Amazon’s business and its workforce. We are looking for a creative thinker who can combine a strong economic toolbox with a desire to learn from others, and who knows how to execute and deliver on big ideas.Ideal candidates will own key inputs to all stages of research projects, including model development, survey administration, experimental design, and data analysis. They will be customer-centric, working closely with business partners to define key research questions, communicate scientific approaches and findings, listen to and incorporate partner feedback, and deliver successful solutions.
Applied Scientist, AWS Proactive Security Detective Tooling
US, VA, Arlington
Job summaryAmazon Web Services is looking for world class scientists to join the Proactive Security group within AWS Security. You will join a team of scientists developing advanced techniques to assess and enhance the security of AWS systems. This group will be visible across AWS teams and to Amazon security leadership as a source of innovative techniques to keep AWS secure. You will help expand our ability to automatically fix security issues by synthesizing patches or automatically applying safe changes to running systems.This role can be based in Seattle, Portland, NYC, Cupertino, Washington DC, Boston, or another office with a significant Amazon Security presence. Each day, hundreds of thousands of developers make billions of transactions worldwide on our cloud. They harness the power of Amazon Web Services (AWS) to enable innovative applications, websites, and businesses. The AWS Security team owns security for all of these services offered by AWS, including EC2, S3, Lambda, and more than 150 others. Our Proactive Security team works with builders across AWS to ensure that AWS products are secure. We dive deep into security technologies such as new authentication systems, hardware security components, cryptography, system hardening, and massive-scale audit analysis. We use techniques drawn from research in Automated Reasoning, Program Analysis, Program Synthesis, and Machine Learning to automatically identify security issues or provide assurance that security requirements are met.You will identify, extend, and prototype advanced techniques that can be applied to solve hard problems in security. In some cases, your team will invent new approaches. In others cases, your team will identify and adapt existing approaches. You will work with security experts and software developers to integrate these techniques into AWS Security’s mechanisms for reducing security risk. AWS will look to your team to find the techniques that will radically improve the security of AWS.While you may not be a security expert yet, in this role you will develop a broad and deep understanding of cloud security and security automation. Our team puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here, which is why we aren’t focused on how many hours you spend at work or online. Instead, we’re happy to offer a flexible schedule so you can have a more productive and well-balanced life—both in and outside of work. Our team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member take the next step in their career. Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Key job responsibilities* Rapidly design, prototype and test advanced security analysis and repair techniques in a high-ambiguity environment, making use of both quantitative and business judgment.* Collaborate with software engineers and security engineers to integrate successful experiments into large scale production services.* Report results in a scientifically rigorous way.* Interact with security engineers and related domain experts to dive deep into the types of challenges that need innovative solutions.* Track relevant research in other Amazon science teams and in the broader research community.