A Hybrid DL and GM (cont’d)+

Applications in Computer Vision

Kayhan BatmanghelichMingming Gong

1

Outlines

• Joint deep feature extraction and learning

• Variational Auto Encoder (VAE)

• Generative Adversarial Networks (GANs)

2

Recap: Variational Auto Encoder (VAE)

3

Data code-length Hypothesis codeStochastic encoder

Encoder!"($|&)

Data x

Amortization(sharing parameters)

max�,✓

F(x, q�) = Eq�(z) [log p✓(x|z)]�KL [q�(z)||p(z)]<latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit><latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit><latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit><latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit>

z ⇠ q�(z|x)<latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit><latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit><latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit><latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit>

Example:

Recap: Mapping the Distributions

4

• VAE learns a mapping that transforms a simple distribution, q(z), to a complicate distribution, q(z|x) so that the empirical distribution (in the x-space) are matched.

Fig Credit: Kingma’s thesis

Recap: Reparametrization Trick

5

Can we improve the estimation of gradient?

(reduce the variance)

g(·)<latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit><latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit><latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit><latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit>

z ⇠ p�(z)<latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit><latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit><latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit><latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit>

Let’s revisit the change of variables:

Recap: How to use this technique in a Graphical Model?

6

Example: Mixture Model

M. Johnson, Composing graphical models with neural networks for structured representations and fast inference, https://arxiv.org/pdf/1603.06277.pdf

✓<latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit><latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit><latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit><latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit>

yn<latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit><latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit><latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit><latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit>

zn<latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit><latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit><latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit><latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit>

�<latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit><latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit><latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit><latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit>

Limitations of VAE• When applied to image data, it results into

blurry images.• For images, it is sensitive to irrelevant

variance, e.g., translations.• Not applicable to discrete latent variables.• Limited flexibility: converting the data

distribution to fixed, single-mode prior distribution

7

GAN: GENERATIVE ADVERSARIAL NETWORK

8

GANs

• Introduced by Goodfellow et al., 2014• Assumes implicit generative model• Asymptotically consistent (unlike

variational methods)• No Markov chains needed• Often regarded as producing the best

samples (but,… no good way to quantify it)

9

Generator Network

10

z ⇠ N (0, I)<latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit><latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit><latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit><latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit>

• Maps noise variable to the data space• Must be differentiable but no invertibility is

required

1-D Example

11Figure: Roger Grosse

Learning

• Train ! to discriminate between training examples and generated samples

• Train " to fool the discriminator

12Figure courtesy: Kim’s slides

Minimax Game

Fixing G

13

Optimal D:

Taking derivative of !log % + 'log(1 − %)

Discriminator Strategy

14

Minimax Game

Substituting

15

GANs minimizes the Jensen–Shannon divergence between two distributions.

More Stable Version

16Redford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”

• DCGAN: Most “deconvs” are batch normalized

17Redford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”

Practical Issues

18

Strong intra-batch correlation

Practical Issues

19

Mode Collapse

Practical Issues

20

Summary• Does not need a specific likelihood function at the last layer to

train a latent-variable model• Similar move from Exact inference to MCMC to var. inf: Don’t

restrict model to allow easy inference - just let a neural network clean up after.

• Topics of research:– How to compute P(z|x), like VAE, without compromising the

quality of samples?– How to make GANs more stable: various divergences, … ?– How to handle discrete inputs ?– How to generalize to idea to GM. Notable works ?

• “Averserial Variational Bayes”, Sebastian Nowozin, Andreas Geiger, 2017 • “Learning in Implicit Generative Models,” Shakir Mohamed, Balaji

Lakshminarayanan, 2016 • “Variational Inference using Implicit Distributions,” Ferenc Huszar, 2017• “Deep and Hierarchical Implicit Models.” Dustin Tran, Rajesh Ranganath,

David Blei, 2017

21

Applications in Vision

Mingming Gong

22

�������

• Semantic Segmentation

• Image-to-Image Translation

23

24

• Partition an image into semantically meaningful regions

25

26

27

Krähenbühl, Philipp, and Vladlen Koltun. "Efficient inference in fully connected crfs with gaussian edge potentials." Advances in neural information processing systems. 2011.

28

• Model long-range connections

• Inference is computationally hard

29

• Label ! = X$,… , X'

• Image ( = I$, … , I'• Gibbs distribution

• Energy function

30

• Label ! = X$,… , X'

• Image ( = I$, … , I'• Gibbs distribution

• Energy function

Unary Pairwise

31

• Unary !"($%)• Computed independently for each pixel• Multinomial logistic regression, boosting

32

• Pairwise potential

!" #$, #& = ((#$, #&)∑,-./ 0 , 1 , (2$, 2&)

• ( - label compatibility function

• k – linear combination of Gaussian kernels

1 , 3$, 3& = exp(−12 2$ − 2& Σ , 2$ − 2& )

33

!" #$, #& = ((#$, #&)∑,-./ 0 , 1 , (2$, 2&)

• ( - label compatibility function

• k – Appearance and local smoothness

( #$, #& = [#$ ≠ #&]Potts Model:

34

https://www.cs.cmu.edu/~efros/courses/LBMV12/crf_deconstruction.pdf

35

• Mean field approximation• Minimize KL divergence !(#||%),

s.t. # ' =)*#*('*)

• Iterative update equation

• Message passing from all '+ to all '*

36

http://vladlen.info/papers/densecrf-slides.pdf

37

• Downsample

38

• Downsample

• Blur the sampled signal

39

• Downsample

• Blur the sampled signal

• Upsample to reconstruct

40

�����������

41

• MSRC dataset• 591 images• 21 classes

42

• PASCAL dataset• 1928 images• 21 classes

43

• CRF can refine the results from unary classifiers

• Limitation: the unary classifiers trained on handcrafted features

• Solution: Deep convolutional nets for dense feature extraction

44

��������

45

46

���� ���������

47

48

• PASCAL dataset• 10, 582 images• 21 classes

�������

• Semantic Segmentation

• Image-to-Image Translation

49

50

51

• Multi-output regression

• ! " − $ %

?

52

z

Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." arXiv preprint (2017).

53

���z

54

55

56

Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434 (2015).

57

����������

58

59

60

P(X, Y) P(X) P(Y)

P(Y|X) Infinite solutions!!

… Discriminator D": #$%& ' ( , *Real zebras vs. generated zebras ……

http://people.eecs.berkeley.edu/~junyanz/talks/pix2pix_cyclegan.pptx

Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint arXiv:1703.10593 (2017).

Discriminator D": #$%& ' ( , *Real horses vs. generated horses

Discriminator D+: #$%& , * , (Real zebras vs. generated zebras ……

mode collapse!

Discriminator

x G(x)

DGenerator

G Real!

Discriminator

x G(x)

DGenerator

G Real too!

x G(x)

Generator

GD

No input-output pairs!

Cycle-consistency LossForward cycle loss: F G x − x %G(x) F(G x )x

Cycle-consistency LossLarge cycle lossForward cycle loss: F G x − x %G(x) F(G x )x Small cycle loss

Cycle-consistency LossBackward cycle loss: ! " # − # %Forward cycle loss: F G x − x %G(x) F(G x )x F(y) G(F x )#

���������

GAN loss:

Cycle consistent loss:

Autoencoder

Cezanne Ukiyo-eMonetInput Van Gogh

Failure cases

74

……

Constrain !" and !#

M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In NIPS, pages 469–477, 2016. 3, 5

75

����

76

……

Infer z from x, VAE, ALI

Ming-Yu Liu, Thomas Breuel, Jan Kautz, "Unsupervised Image-to-Image Translation Networks" NIPS 2017 Spotlight, arXiv:1703.00848 2017

����

77

����

78

!"##→% = '%()# ∼ +# )# "# )

����

79

����

80

����

81

82

�������

83

�������

84

�������

85

Hoffman, Judy, et al. "CyCADA: Cycle-Consistent Adversarial Domain Adaptation." arXivpreprint arXiv:1711.03213 (2017).

86

87

88

�������